All Ruckus Unleashed AP's Lose Heartbeat Simultaneously

Issue:

AP (name@MAC) joins with uptime (xxxx) s and last disconnected reason (Heartbeat Loss)

1. ALL Unleashed AP's lose heartbeat within seconds of each other

2. Occurs at a regular interval

3. AP uptime similar

DHCP lease time may be the issue.

Detail:

I was called in to help with an Unleashed wireless network. The customer was a winery with a B&B. They would complain of connection issues at check-in time (4 PM local - Pacific). The customer assumed there was a problem with the wireless network relating to many new devices attempting to connect around the same time. They instructed their customers to wait until 4:30 to try to connect to the Wi-Fi.

The system logs showed that all AP's (19) would lose heartbeat twice a day, at the same time every day, Noon and Midnight. The drops occurred every day as far back as the logs went, 30 days. The timing didn't match the customer report of four PM.

Looking at the NTP settings, the system was using GMT. Pacific time is -8, so Midnight = 4 PM. We are getting somewhere. All AP's go down at 4 PM every day, 4 AM also but no complaints at that time.

All AP's lost heartbeat within about 40 seconds: these AP's were spread across 5 PoE switches. It wasn't likely to be power related. The customer didn't report any daily sitewide power events.

What happens every 12 hours on this network? DHCP lease renewal? I didn't have access to LAN documentation, so how do I find the DHCP server IP and how long the lease is?

Find DHCP server and lease time:

Using a MAC

View the contents of /var/db/dhcpclient/leases/en0.plist (replace 'en0' with your wireless NIC)

> sudo su

> cd /var/db/dhcpclient/leases

> sh-3.2# cat en0.plist

<?xml version="1.0" encoding="UTF-8"?>

<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"

"http://www.apple.com/DTDs/PropertyList-1.0.dtd">

<plist version="1.0">

<dict>

<key>ClientIdentifier</key>

<data>

AcSRDLHt7w==

</data>

<key>IPAddress</key>

<string>192.168.0.120</string>

<key>LeaseLength</key>

<integer>43200</integer> [<---- = 12 hours]

<key>LeaseStartDate</key>

<date>2020-12-08T16:00:02Z</date>

<key>PacketData</key>

<data>

AgEGAMJT9m8AAAAAAAAAAMCoAHgAAAAAAAAAAMSRDLHt7wAAAAAAAAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABjglNjNQEFNgTAqAABMwQAAKjA

AQT///8AAwTAqAABBgTAqAAB/wAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

</data>

<key>RouterHardwareAddress</key>

<data>

1GqReXYU

</data>

<key>RouterIPAddress</key>

<string>192.168.0.1</string>

<key>SSID</key>

<string>Welcome_Inn</string>

</dict>

</plist>

-OR-

Run the following command

> ipconfig getpacket en0 (again substitute your wireless NIC here)

op = BOOTREPLY

htype = 1

flags = 0

hlen = 6

hops = 0

xid = 0xc253f66f

secs = 0

ciaddr = 0.0.0.0

yiaddr = 192.168.0.120

siaddr = 0.0.0.0

giaddr = 0.0.0.0

chaddr = c4:91:c:b1:ed:ef

sname =

file =

options:

Options count is 7

dhcp_message_type (uint8): ACK 0x5

server_identifier (ip): 192.168.0.1 [<---- DHCP server]

lease_time (uint32): 0xa8c0 [<---- hex value converts to 43200 seconds = 12 hours]

subnet_mask (ip): 255.255.255.0

router (ip_mult): {192.168.0.1}

domain_name_server (ip_mult): {192.168.0.1}

end (none):

Bingo! 12 hour lease expiration. I rechecked the logs to confirm IP addresses. In this environment, when Unleashed AP's re-initiate DHCP lease, they are given a new IP address.

A packet capture of a DHCP lease renewal would also work:

Frame 2 - DHCP server (192.168.0.1) returns a DHCP offer with client IP address (192.168.0.3) & lease time (1 minute)

Solution:

When the Unleashed master AP is not at the IP address that it is 'supposed' to be, all other AP's lose their connection and report heartbeat loss. Eventually, the network heals itself. In this case, the entire process took about five minutes. Five minutes of network downtime.

Reconfiguring all AP's to manual IP solved the issue.

--WirelessKahuna