Wednesday, August 24, 2016

Why validation surveys aren't enough, the importance of functional validation testing.

I've been thinking a lot lately about post validation surveys and what value they provide.  And I'm starting to wonder if the premise of validation surveys isn't being realized. After all, what exactly are you validating?

I've seen surveys that looked great, -67dbi coverage, 30 SNR,  no CCI, no interference, and yet those site have been the receiving complaints about the wifi.  A visit to the site illuminates some challenge that the survey didn't convey, and usually easily remedied.  Sometimes it's RF related. Sometimes configuration.

Surveys don't lie, but they tell the story from their perspective.  Let's call this subjective reality.  And the reality that your survey rig sees may vary dramatically from what other devices see, so much that it appears false.  Think of a cog native bias around how Wi-Fi works as seen by the device testing it.

There has been a lot written on "compensation" lately and how we can compensate the delta from the survey device to the real device. But I don't know if this can overcome enough of the differences between our survey device and the variety of devices out there.  Sure, I can compensate a fixed amount from my survey NIC to the actual device.  Is that compensation value flat?  Or is there a compensation value per channel?

And then let's talk about noise, SNR, overlap, roaming, etc.  Can we compensate for all of these variables per channel?  Suddenly compensation seems about as practical as calibration.  Then we have another view to consider: how the APs see things.  So how do we validate networks?  

Functional Validation Testing:
Personally, I think engineers spend too much time focusing on surveys and not enough time validating that the network meets the requirements for the devices using it.  That means getting the actual devices on the network and testing that they meet the requirements.  Who's requirements?  The business requirements for their devices.

Ever seen a device that fails to roam if it sees more than one "good" candidate?  Those are fun.  What about an iPhone that can see a network but won't automatically associate to it even when configured to?  But... But... But... The survey results look fine.  You won't find those problems on a Windows machine running your favorite survey software.

Take that forklift and barcode scanner for a ride and make sure it roams and works they way it's intended.  Spend a day walking the facility measuring voice call quality, roaming, throughput, etc with the devices that will actually be deployed.  This ensures that you know and understand the device, how it will be used, and identify places where we may have issues.  Have actual workers show you how they use it.

This process also makes you think about how you can validate if the device works normally.  Maybe it's packet captures looking for L2 retransmission?  Measured roaming delays?  Maybe it's constantly scanning a barcode into the order management system as you fly by on a forklift.  Maybe it's wired side packet captures looking for Telnet or SSH sessions dropping and reconnecting within a specific amount of time.  Or, if you are lucky, it might be putting the device into a survey mode and see how t sees the network around it.

Spend the day after the turn-up working onsite.  Find a comfy chair and work wirelessly for the day.  Take every device you can manage with you.  Make sure they behave as you expect them to.  See odd behavior?  End users will too.

Now, before I get a bunch of angry emails, I'm not saying validation surveys are dead.  I'm saying that we need to spend more time validating the actual devices and less about a relative measurement that may or may not be indicative of a how well a network performs for specific devices.  Validation surveys give us a relative baseline from which to judge a network, which is valuable.  It does not however tell us how well a network functions.

As my friend Chris Lyttle ( so eloquently put it: "It's a system, all things matter."  Validating the RF is one thing, but testing the rest of the system is just as important.

Thursday, August 11, 2016

CMX Cloud with Mobility Express is here

For some, Mobility Field Day 1 may be a distant memory.  The first Mobility Field Day Cisco brought Cisco to the delegates to talk about CMX Cloud.  Simply taking presence analytics to the cloud wasn't all that interesting to me.  Prior to 8.3 code you needed a proxy server to convert the NMSP data from the controller to an HTTPS connection to the CMX server in the cloud.  This was a bummer for me because now I have to deploy a server to run the proxy on, as well as a WLC to connect to CMX.   And if I'm deploying a server infrastructure at a site, why not just run CMX locally?

But not all was lost, Cisco told us the plan was for 8.3 to support a native connection to CMX cloud.  So that removes the server requirement, but what about the WLC?  They also announced that Mobility Express (the WLC software running on an AP) would support this direct connection to CMX Cloud in 8.3, removing the requirement to have a physical WLC and controller licensing for this deployment.  This is great news for distributed organizations like retail that want to gain presence analytics, but are shy about either deploying gear locally, or building/leveraging their WAN circuits for this.

Fast forward to today and 8.3 code for Mobility Express is published on  I was asked to build a CMX demo for the upcoming Interface Boise conference in a few weeks, so I decided what a great opportunity to try out CMX Cloud with Mobility Express.  In just a few minutes I had reset my 1832 AP running Mobility Express and run through the initial setup.  A reboot later and the Mobility Express AP was up and operational.

Now for the CMX Cloud component.  I signed up for a CMX Cloud trial at  It warns you that it can take up to a day to receive your account info.  Less than an hour later, I had my login credentials, server url and authentication token.   Going to the Mobility Express GUI, I was able to configure this with a minimal effort.

*Note: I ran into an issue where during the Mobility Express initial setup, DNS was listed as optional.  When you setup CMX, it NEEDS the DNS server functional to resolve the CMX server URL and to build the needed ACLs for the CMX Connect functionality.  You can configure DNS on the CLI of the Mobility Express if you don't want reset the AP and rerun the setup.  Once DNS was setup, the CMX configuration on the AP worked flawlessly.

If you're familiar with CMX Presence, the setup of CMX Cloud for presence is quite straight forward.  First you create a site and add your AP to CMX.  Remember it's the base radio MAC, not the ethernet MAC.  

Once you have the site created and the AP added to the site, I verified that I saw the Mobility Express controller connected to CMX and that I started to receive data.

Now that the AP is talking to CMX Cloud, for the next part all I had to do was wait (for data to populate).
The AP was running at my desk in the CompuNet Meridian office.  Pretty quickly i started to see the Insights panes starting to get populated.

I found the insights pretty interesting.  Our local engineering team is still edged out on Intel PCs vs Macs, but it's close (51 to 46 in the last 7 days).  I did find it interesting that 12-1 was he peak hour, considering the number of people who go out for lunch.

All things considered, this solution looks really well thought out, easy to deploy and simple from the fact that the only hardware required on-site is the Mobility Express AP.  I’m also really interested to see what this data looks like for the day of the conference (which is the whole point of the demo).

For the future, I think having location as part of this offering would be cool.  However, I think that the need to have maps in Prime Infrastructure and sync that to CMX might be more cumbersome and really take away from the minimal hardware needed.  Maybe this is an opportunity for Prime Infrastructure in the cloud (Or a “Prime Lite” in the cloud).  I also like the fact that I don’t have to manage the box.  It has been exceptionally stable and not having to update it periodically exemplifies why this belongs in the cloud.  This is really attractive when you realize that Cisco markets this to non-IT folks.  IT Doesn’t have to support it aside from 3 lines of config in the Mobility Express AP, and the rest can be managed by the marketing or BI folks.

It should be noted that Cisco provided me with the 1832 access point as part of participating in previous Wireless Field Day activities.  The CMX Cloud was provided by Cisco as part of a 60 day trial.