And shortly after starting on my testing, things started to go horribly wrong. I had gotten too aggressive with the setting and my test clients had "hit the wall." Meaning that my clients dropped below the RX-SOP threshold and the AP was no longer responding to it.
Not only did this impact my client performance, but roaming became exponentially worse. I had discovered the "golden rule" of RX-SOP: Do not interfere in the natural roaming of clients or alter a APs effective cell size through RX-SOP.
A lot of the mystery around RX-SOP is that how it should be used isn't well documented. Therefore people look at the classic Cisco presentations about applying earmuffs to an AP and make the assumption that it is about creating smaller cells. Which it is not.
RX-SOP is meant to reduce an AP's RX contention domain by tuning out CCI from adjacent cells on the same channel. That "adjacent cell" part is really important, because we don't want to impact clients in the AP's effective cell.
The idea is that if we hear an adjacent cell at -79, we will defer to transmissions from that cell. RX-SOP lets us set our threshold higher than neighboring APs and cease differal to those adjacent cells.
So why don't we want to change cell size with RX-SOP? Mostly because it is a 1-way alteration to the cell. There is no way to let the client know he's hit that lower boundary and therefore clients cannot take that into account in their roaming algorithm. Also, clients can vary their TX power, making the clients RSSI at the AP varied. Since client behavior on this front is so inconsistent, it's best to avoid impacting the local cell with this feature.
Let's walk through an example of trying to impact an AP's cell size with RX-SOP and show why it doesn't work. Let's set our RX-SOP threshold at -72 on 2.4Ghz, start a large file transfer and start walking away. We rate shift normally as we walk away until the clients signal at the AP drops below -72. At which point the client is no longer being ACK'd by the AP.
Immediately we see the client start to rate shift more aggressively due to packet loss. Most clients drop quickly from high MCS rates down to the lowest supported rates. At which point the client realizes even though it is still hearing probes that the AP is gone and begins panic probing (probing aggressively looking for the next AP to roam to).
The golden rule keeps us out of this situation. There are other mechanisms to change the AP's cell size. Primarily TX power, supported data rates are methods to control cell sizing. More modern features such as Optimized Roaming with 11k and 11v can also effectively reduce cell sizing.
In terms of implementing RX-SOP, one of the recommendations is to take the weakest client in the cell(s) and subtract 10db. So if over time you have the weakest client at -70, you would use -80 as your RX-SOP threshold.
I can't stress this enough, but you really need to design your network to use RX-SOP. If your margin from the weakest client to your neighboring cell that you want to filter out is less than 10dB, you either won't get the benefit of tuning out that adjacent cell, or you risk letting stations hit the wall.
The use of directional antennas helps with this as you increase RX reception of clients via the antenna gain, and you reduce the signal from adjacent cells. This can widen the gap between client and adjacent cells allowing more moderate RX-SOP thresholds with less risk of impact.
Also RX-SOP is one of the last adjustments I make. Changing power levels, channels, and data should probably occur first. Remember, you are not tuning the current cell, but tuning out adjacent cells. If you start to impact the current cell, you've gone too far.