Too Much of a Good Thing
I was brought into a site by an AP manufacturer to help with an issue their customer was having. The customer complained about interference and performance and was about to replace everything.
I was given network documentation from the vendor SE who did the network design (no charge no site visit). At first glance, it seemed like an aggressive number of access points. The SE for the manufacturer insisted that there were "special circumstances" with this customer. I was skeptical but intrigued.
In the first meeting, the customer made it very clear that their situation was unique. Their product is an IP video and telephony solution, and they "require more Wi-Fi than normal," this raised a few alarms. After the original installation, the customer identified some areas as "dead spots," so they added a few more access points. The additional AP's didn't seem to help, so they figured the problem must be with the new wireless LAN equipment.
Shortly after the WLAN was installed, several electric vehicle charging stations were installed in an adjacent parking lot. Additionally, there was a cell tower on the roof of the building. The client suspected both of these were massive sources of Wi-Fi interference.
No amount of discussion would convince them that neither of these was likely to be interfering with their WLAN. So, I performed passive and spectrum analysis surveys near both the charging stations and the cellular antennas. While the charging stations did have a Wi-Fi network, they were far enough away from the customer's property that they weren't likely to cause any interference.
A passive site survey inside the customer's building showed the charging station SSID barely registering at -90 dBm and only near a single window. I informed the customer that this network wasn't likely to trigger the clear channel assessment's carrier sense threshold, so this was not a source of interference. As expected, nothing showed up in either 2.4 or 5 GHz from the cell tower in either the passive or RF surveys. I was able to demonstrate this with heat maps visually. As it turns out, a picture (or a heatmap) is worth a thousand words.
The in-building site survey showed two primary issues in 5 GHz. First, there was massive cochannel interference (CCI). Second, the access points were not all broadcasting the same SSID's. While this is not necessarily a bad thing, at this site, it was. The expectation was that all SSID's would be available everywhere. Additionally, no non 802.11 RF interference was found.
There were a few contributing factors to the CCI. The wireless LAN controller (WLC) was configured to allow only UNII-1 and UNII-3 channels. Additionally, these channels were 80 MHz wide, and there were a lot of radios for the space. When assigning AP channels, the WLC did not avoid secondary channels. On top of this, all APs were set to transmit at full power. I found the source of their interference. The good news was it was going to be an easy fix.
Heat maps showed that there were, in effect, "dead" spots. However, it wasn't for lack of RSSI. For no apparent reason, there were six different AP groups. There didn't appear to be any physical or logical reason to create groups at all. These groups were not configured to broadcast SSID's consistently, resulting in coverage heatmaps per SSID that were very different. In other words, dead spots.
If possible, the 2.4 GHz band was even worse. All eleven 2.4 GHz channels were in use. All access points were transmitting at full power. 2.4 GHz was all but unusable.
All remediation recommendations were modeled in Air Magnet and approved by the customer before making any changes. The fix for the 5 GHz band was to strategically remove a third of the access points, use 20 MHz channel widths, decrease transmit power across the board, and enable the UNII-2a and UNII-2c DFS channels.
To fix the 2.4 GHz band, I configured the WLC to allow only channels 1, 6, and 11. I reduced transmit power in the remaining AP's and disabled a few more of the 2.4 GHz radios. There were very few 2.4 GHz client devices, so a coverage design in this band was acceptable.
After making these changes, per client throughput increased dramatically. Packet loss and retransmissions decreased significantly, trouble tickets dropped to near zero, and there were no more issues with the customer's "proprietary application." The customer was now happy with their wireless network. So much so, they flew me to their other offices to remediate their other wireless networks.
With the right tools and experience, I find about 90% of WLAN troubleshooting issues can be identified within the first five minutes of an engagement. The rest may take some work.
Lessons learned:
• A picture or a heatmap is worth a thousand words.
• If the wireless network design is free with no onsite calibration, it's probably worth what you paid for it.
• Don't blindly trust controller default config - 80 MHz channel width - UNII-1 and UNII-3 only - all 2.4 GHz channels in use - all AP's running at full transmit power is not likely the way one should intentionally configure a wireless network. (you probably know which vendor this is)
• Know what your configuration changes will do and test them. Don't assume the configuration is performing as expected. Survey the environment to determine what is happening.
Most importantly, no wireless network is any more "special" than any other. Each one will be unique in many ways; however, Physics isn't concerned about "special circumstances." As design engineers, all we have to work with are client device capabilities, WLAN equipment capabilities, and RF characteristics of the space. Everything must balance.
Kahuna-Fi
Mike Wade
@WirelessKahuna