Upload
others
View
5
Download
0
Embed Size (px)
Citation preview
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
1
© 2008 Cisco Systems, Inc. All rights reserved. Cisco PublicBRKAGG-301114511_04_2008_c1 2
Troubleshooting Wireless LANs with Centralized Controllers
BRKAGG-3011
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
2
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 3BRKAGG-301114511_04_2008_c1
Troubleshooting Wireless LANs
Technology Refresher
Set up Your Network Right
Make Sure Stuff Basically Works
Get Individual Trouble Spots Fixed Up
Get Individual Clients Fixed Up
Nothing to It!
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 4BRKAGG-301114511_04_2008_c1
Technology Refresher
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
3
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 5BRKAGG-301114511_04_2008_c1
Wireless LAN technology refresher
802.11/802.1X/WPA
Cisco Unified Architecture/LWAPP
Cisco Unified client mobility
Radio Resource Management
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 6BRKAGG-301114511_04_2008_c1
WLAN Topologies – Single 802.11 AP
STA 1
STA 3STA 2
Channel 11
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
4
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 7BRKAGG-301114511_04_2008_c1
WLAN Topologies—InfrastructureMultiple cells
Each cell operates on its own channel
Each AP transmits beacons advertising its BSSID (radio MAC)
All APs offer the given service using the same ESSID (“SSID”)
channel 11channel 6
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 8BRKAGG-301114511_04_2008_c1
Steps To Building an 802.11 Connection
1. Listen for Beacons
2. Probe Request
3. Probe Response
4. Authentication Request
5. Authentication Response
6. Association Request
7. Association Response
8. (Optional: EAPOL authentication)
9. (Optional: encrypt data)
10. Move user data
State 1:Unauthenticated,
Unassociated
State 2:Authenticated,Unassociated
State 3:Authenticated,
Associated
802.11 assoc complete
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
5
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 9BRKAGG-301114511_04_2008_c1
802.11 Association Overview (with WPA IE)Station Access Point
Beacon (WPA-IE)
Probe Response (WPA-IE)
Authentication Response
Association Response
Probe Request
Association Request (WPA-IE)
Authentication Request
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 10BRKAGG-301114511_04_2008_c1
802.1X authentication – dynamic WEPServer
EAP-ID-Request
Rest of the EAP Conversation
Radius-Access-Accept
(key)EAP-Success
EAPOL-START
EAP-ID-ResponseRADIUS (EAP-ID_Response)
SupplicantAuthenticator
The Supplicant Derives the Session Key from User Password or Certificate and Authentication Exchange
Session Key
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
6
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 11BRKAGG-301114511_04_2008_c1
EAP-FAST Authentication Overview
RADIUS server
EAPOL Start
EAP-Request/Identity
EAP -Response/Identity (EAP-ID)
EAP success
RADIUS Access request
Start EAP AuthenticationAsk client for identity
Access Request with EAP-ID
Perform sequence defined by EAP-FAST
Client derives PMK
keykeyRADIUS Access Accept
(Pass PMK to AP)
Supplicant
EnterpriseNetwork
Secure Tunnel (via TLS & PAC)
Client-side Authentication
AP
WPA Key Management
Protected DATA Transfer
Phase 1
Phase 2
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 12BRKAGG-301114511_04_2008_c1
Review: Cisco’s Unified Architecture
Cisco Centralized WLAN Model
Split MAC and Local MAC
LWAPP Architecture (Layer 3 LWAPP)(Layer 2 LWAPP is going away)
Mobility—Layer 2 and Layer 3
Radio Resource ManagementDynamic Channel Assignment (DCA)
AutoRF
Coverage Hole Detection (CHD)
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
7
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 13BRKAGG-301114511_04_2008_c1
Cisco Centralized WLAN Model
Ingress/Egress Point from/to Upstream
Switched/Routed Wired Network (802.1Q Trunk)
Control MessagesData Encapsulation
Access Points Are “Lightweight”—Controlled by a Centralized WLAN Controller
Much of the Traditional WLAN Functionality Moved from Access Points to Centralized WLAN Controller
LWAPP Defines Control Messaging and Data Encapsulation Between Access Points and Centralized WLAN Controller
Lightweight Access Point
Wireless LAN Controller
LWAPP Tunnel
Switched/Routed Wired Network
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 14BRKAGG-301114511_04_2008_c1
Cisco Centralized WLAN Model
Ingress/Egress Point from/to Upstream
Switched/Routed Wired Network (802.1Q Trunk)
Switched/Routed Wired Network
Control MessagesData Encapsulation
Lightweight Access Point
Wireless LAN Controller
Remote RF Interface
Real-time 802.11 MAC
RF Spectral Analysis
WLAN IDS Signature AnalysisSecurity ManagementQoS Policies EnforcementCentralized Configuration, Firmware ManagementNorthbound Management Interfaces
LWAPP Carries All Communication Between Access Point and ControllerL2 or L3 TransportMutual Authentication—X.509 Certificate BasedLWAPP Control AES-CCM EncryptedData Encapsulation
Radio Resource Management
Mobility Management
LWAPP Tunnel
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
8
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 15BRKAGG-301114511_04_2008_c1
Division of Labor—Split MAC
Ingress/Egress Point from/to Upstream
Switched/Routed Wired Network (802.1Q Trunk)
Switched/Routed Wired Network
Control MessagesData Encapsulation
Lightweight Access Point
Wireless LAN Controller
Real-time 802.11/MAC Functionality:Beacon GenerationProbe ResponsePower Management/Packet Buffering
Data Encapsulation/De-encapsulationFragmentation/De-fragmentation
802.11e/WMM Scheduling, QueueingMAC Layer Data Encryption/Decryption802.11 Control Messages
Non Real-Time 802.11/MAC Functionality:Association/Disassociation/Reassociation802.11e/WMM Resource Reservation
802.11 Distribution ServicesWired/Wireless Integration Services
802.1X/EAPKey Management
LWAPP Tunnel
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 16BRKAGG-301114511_04_2008_c1
Division of Labor—Split MAC Illustrated
802.11 Beacon
Probe Request
Probe Response
802.11 Authentication/Association
802.1X Authenticationand 802.11i Key Exchange
802.11 Data
Probe Is Processed by the AP and Forwarded to
the Controller
802.11 Action Frames
Encryption/Decryptionof RF Packets
Handled at the AP
Add Mobile(AES-CCMP, PTK)
LWAPP Tunnel
Add Mobile(Cleartext, 802.1X Only)
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
9
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 17BRKAGG-301114511_04_2008_c1
Layer 3 LWAPP Architecture
Access points require IP addressing
APs can communicate with WLC across routed boundaries
L3 LWAPP is more flexible than L2 LWAPP and all products support this LWAPP operational ‘flavor’
Ingress/Egress Point from/to Upstream
Switched/Routed Wired Network (802.1Q Trunk)
Data Encapsulation—UDP 12222Control Messages—UDP 12223
Lightweight Access Point
Wireless LAN Controller
LWAPP Tunnel
Layer 2/3 Wired Network—Single or Multiple Broadcast Domains
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 18BRKAGG-301114511_04_2008_c1
Layer-2 Roaming—Inter-Controller
Client must be re-authenticated and new security session established
Client database entry moved to new controller
No IP address refresh needed
L2 Inter-Controller roam happens when an AP moves association between APs joined to the different controllers but client traffic bridged onto the same subnet
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
10
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 19BRKAGG-301114511_04_2008_c1
Layer-3 Roaming—Inter-controller
L3 Inter-Controller roam happens when an AP moves association between APs joined to the different controllers but client traffic bridged onto different subnet
Client reauths
Client database entry copied to new controller
Original controller is “anchor”; new one is “foreign”
EoIP tunnel automatically established
No IP address refresh needed
Asymmetric traffic path established – or
Symmetric traffic path
“Anchor”
“Foreign”
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 20BRKAGG-301114511_04_2008_c1
Radio Resource Management refresher
Dynamic Channel Assignment (DCA)Selects channels for the radios to use
Responds to interference
AutoRF (Dynamic Power Control, DPC)Reduces radio power, to ensure that each radio hears exactly 3 others at or above the tx-power-thresh value
Coverage Hole Detection (CHD)Detects “coverage holes”, by identifying clients from which we are receiving a poor signal, and accordingly increases radio power, to compensate
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
11
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 21BRKAGG-301114511_04_2008_c1
Set up Your Network Right
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 22BRKAGG-301114511_04_2008_c1
Build out your infrastructure
Use the right wired network
RRM tuning tips
bonus! WLC Config Checker
Get your APs to join
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
12
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 23BRKAGG-301114511_04_2008_c1
Wired Network Requirements—AP to WLC (LWAPP Path)
Network RTT - <= 100msec, bandwidth >= 128kbps
Network path must be able to pass IP fragments (but never generate a fragment < 32 bytes)
Network path must not deliver IP fragments via multiple links
APs can be NATted, but WLCs cannot
Trust/set QoS marking as needed for voice
LWAPP
AP WLC
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 24BRKAGG-301114511_04_2008_c1
LAG Can’t Reassemble Fragments from Multiple Ports
No IP fragments < 32 bytes (CSCsh96186)
All fragments of any IP datagram must arrive on the same port
src-dst-ip is recommended
src-dst-port will lose
Network must not load balance packets into different LAG ports
4404 subsystem
Link Aggregation
Bundles
4404 subsystem
WiSM
4404
Link AggregationBundle
4402
Link aggregation bundle
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
13
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 25BRKAGG-301114511_04_2008_c1
Wired Network Requirements—WLC to WLC Mobility Tunnel (EoIP Path)
NAT can be used in the EoIP path (as of 4.2.61.0)
Network path must be able to pass 1500B IP packets unfragmented(CSCsm05607)
Workaround: configure wireless devices to use a small MTU
Partial workaround: use ip tcp adjust-mss
Internet
DMZ Corporate Network
Ethernet in IP Tunnel
LWAPPEncapsulation
LWAPPEncapsulation
ip tcp adjust-mss 1300
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 26BRKAGG-301114511_04_2008_c1
Radio Resource Management - autoRF
config advanced 802.11b tx-power-control-thresh is the master fader for radio power (values in -60 to -80dBm—lower values for denser installations)
thresh-68
thresh-73
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
14
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 27BRKAGG-301114511_04_2008_c1
Radio Resource Management – detune CHD
Detune Coverage Hole Detection if too many APs are at power 1 in a dense environment (“sticky client” problem)
shrink Coverage threshold (e.g. to 6 dB)
boost Min Clients (e.g. to 5)
See “Radio Resource Management under Unified Wireless Networks”, Document ID 71113, cisco.com
I can’t hear this client too well –
better boost my power!
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 28BRKAGG-301114511_04_2008_c1
WLC Config Checker
Windows GUI program, analyzes the output of show run-config
Use config paging disable
Console at 115200 bps, or telnet/ssh
Try to hit return right away when prompted
Provides warnings about the configuration
Displays key aspects of the configuration, and of AP RF info
On CCO, in wireless software downloads area [?]
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
15
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 29BRKAGG-301114511_04_2008_c1
WLC Config Checker—Warnings
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 30BRKAGG-301114511_04_2008_c1
WLC Config Checker—AP Nearby Info
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
16
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 31BRKAGG-301114511_04_2008_c1
AP Join troubleshooting
First, the AP must Hunt for the IP addresses of possible WLCs to join
Next, the AP sends Discover messages to all the WLCs, to find out which ones are alive
Then the AP picks the best WLC and tries to Join it
For details, see “Lightweight AP (LAP) Registration to a Wireless LAN Controller (WLC)”, Document ID 70333, cisco.com
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 32BRKAGG-301114511_04_2008_c1
LWAPP AP State MachineAP runs HUNTINGalgorithm to find
candidate controllers to join
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
17
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 33BRKAGG-301114511_04_2008_c1
L3 LWAPP WLC Address Hunting
1. LWAPP Discovery broadcast on local subnetcan use ip helper-address, ip forward-protocol
2. Over-the-Air Provisioning (OTAP)
3. Locally stored controller IP addresses
4. DHCP vendor specific option 43 (IP Address should be “Management Interface” IP)
5. DNS resolution of “CISCO-LWAPP-CONTROLLER.localdomain”(should resolve to the “Management Interface” IP)
6. If no controller found, start over
AP Goes Through the Following Steps to Compile a Single List of WLAN Controllers
Note: The Actual Order of This Process Is Irrelevant Because Each AP Goes Through All Steps Before Proceeding to the Next Phase
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 34BRKAGG-301114511_04_2008_c1
L3 LWAPP WLC Discovery
Discover
Discover
Discover
Discover
X
Discover
AP tries to send Discover messages to all the WLC addresses that its Hunting process turned up
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
18
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 35BRKAGG-301114511_04_2008_c1
L3 LWAPP WLAN Controller Discovery Algorithm
Once a list of WLAN Controllers is compiled, the AP sends a unicast LWAPP Discovery Request message to each of the controllers in the list
WLAN Controllers receiving the LWAPP discovery messages respond with an LWAPP Discovery Response
LWAPP Discovery Response contains important information:
Controller name, controller type, AP capacity, current AP load, “Master Controller” status, AP-Manager IP address
AP waits for its “Discovery Interval” to expire, then selects a controller and sends an LWAPP Join Request to that controller
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 36BRKAGG-301114511_04_2008_c1
WLAN Controller Selection Algorithm
1. If the AP has been configured with primary, secondary, and/or tertiary controller, the AP will attempt to join these first (this is resolved in the Controller “name” field in the LWAPP Discovery Response)
2. Attempt to join a WLAN Controller configured as a “Master” controller
3. Attempt to join the WLAN Controller with the greatest excess AP capacity
The AP Selects the Controller to Join Using the Following Criteria
Note: This Last Step Provides the Whole System with Automatic AP/WLC Load-Balancing Functionality
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
19
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 37BRKAGG-301114511_04_2008_c1
WLAN Controller Join Process—Mutual Authentication – stress SSC
AP LWAPP Join Request contains the AP’s signed X.509 certificateWLAN Controller validates the certificate before sending an LWAPP Join Response
Manufacture Installed Certificate (MIC)—Cisco 1000 Series, all Cisco Aironet APs manufactured after July 18, 2005Self-Signed Certificate (SSC)—LWAPP Upgraded Cisco Aironet APs manufactured prior to July 18, 2005SSC APs must be “authorized” on the WLAN Controller
If AP is validated, the WLAN Controller sends the LWAPP Join Response which contains the controller’s signed X.509 certificateIf the AP validates the WLAN Controller, it will download firmware if necessary and then request its configuration from the WLAN controller
Note: In the Configuration Information, the WLC Includes the IP Addresses of All Other Controllers in Its Mobility List. APs Then Send LWAPP Discovery Messages to Those WLCs, As Well
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 38BRKAGG-301114511_04_2008_c1
Troubleshooting LWAPP-Based APs
Can the AP and the WLC communicate?
Make sure the AP is getting an address from DHCP (check the DHCP server leases for the AP’s MAC address)
If the AP’s address is statically set, ensure it is correctly configured
Try pinging the AP from the controller
If pings are successful, ensure the AP has at least one method by which to discovery at least a single WLC
Console or telnet/ssh into the controller to run debugs
Check the Basics First
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
20
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 39BRKAGG-301114511_04_2008_c1
Successful LWAPP AP Join
(WLC_CLI) >debug mac addr 00:0b:85:54:ce:00
(WLC_CLI) >debug lwapp events enable
Received LWAPP DISCOVERY REQUEST from AP 00:0b:85:54:ce:00 to 00:0b:85:40:4a:c0 on port '29'
Successful transmission of LWAPP Discovery-Response to AP 00:0b:85:54:ce:00 on Port 29
Received LWAPP JOIN REQUEST from AP 00:0b:85:54:ce:00 to 06:0a:20:20:00:00 on port '29'
LWAPP Join-Request MTU path from AP 00:0b:85:54:ce:00 is 1500, remote debug mode is 0
Successfully transmission of LWAPP Join-Reply to AP 00:0b:85:54:ce:00
Register LWAPP event for AP 00:0b:85:54:ce:00 slot 0
Received LWAPP CONFIGURE REQUEST from AP 00:0b:85:54:ce:00 to 00:0b:85:40:4a:cb
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 40BRKAGG-301114511_04_2008_c1
Failed LWAPP AP Authentication
(WLC_CLI)>debug mac addr 00:12:80:ad:7a:9c(WLC_CLI)>debug lwapp events enable [TIME]: Received LWAPP DISCOVERY REQUEST from AP 00:12:80:ad:7a:9c
to ff:ff:ff:ff:ff:ff on port '1'[TIME]: Successful transmission of LWAPP Discovery-Response to AP
00:12:80:ad:7a:9c on Port 1[TIME]: Received LWAPP JOIN REQUEST from AP 00:12:80:ad:7a:9c to
06:0a:10:10:00:00 on port '1'[TIME]: LWAPP Join-Request does not include valid certificate in
CERTIFICATE_PAYLOAD from AP 00:12:80:ad:7a:9c.[TIME]: Unable to free public key for AP 00:12:80:AD:7A:9C [TIME]: DEBU CTRLR spamProcessJoinRequest:1574
spamProcessJoinRequest : spamDecodeJoinReq failed
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
21
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 41BRKAGG-301114511_04_2008_c1
Set the WLC’s Time
Make sure each controller has the correct time set
Check the WLC’s time:(WLC_CLI) >show time
Manually set the time:(WLC_CLI) >config time manual <MM/DD/YY> <HH:MM:SS>
Or, use NTP: (WLC_CLI) >config time ntp server <Index> <IP Address>
(WLC_CLI) >config time ntp interval <3600 - 604800 sec>
The #1 Reason APs Fail to Join Is Inaccurate Controller Time
Note: NTP Is Not a Quick Fix Because It Is Only Invoked at Controller Boot and at the NTP Interval
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 42BRKAGG-301114511_04_2008_c1
Taking Care of SSCs
Each AP has been upgraded properly with the correct time (this date is like your favorite beer’s born-on date)
Each WLC is configured to allow SSC AP authentication
Each WLC has each AP’s SSC hash
Display the input SSC hashes and whether SSC support is enabled:(WLC_CLI) >show auth-list
Enable SSC support:(WLC_CLI) >config auth-list ap-policy ssc enable
Input each AP’s hash:(WLC_CLI) >config auth-list add ssc <MAC Address> <40-bit SSC Hash>
Note: If You’re Not Sure Whether an AP Has Created an SSC, in the AP’s CLI, Do a #show crypto ca certificates. Any Output Indicates that Your AP’s Got Its Certificates
If Controller Time Is Correct, but SSC APs Fail to Join, Make Sure:
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
22
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 43BRKAGG-301114511_04_2008_c1
What If the AP’s SSC Hash Is Missing?(WLC_CLI) >debug mac addr 00:12:80:ad:7a:9c(WLC_CLI) >debug pm pki enable[TIME]: * sshpmGetIssuerHandles:1427 sshpmGetIssuerHandles: locking ca cert table[TIME]: * sshpmGetIssuerHandles:1435 sshpmGetIssuerHandles: calling x509_alloc() for user cert[TIME]: * sshpmGetIssuerHandles:1439 sshpmGetIssuerHandles: calling x509_decode()[TIME]: * sshpmGetIssuerHandles:1458 sshpmGetIssuerHandles: <subject> L=San Jose, ST=California, C=US, O=Cisco Systems[TIME]: * sshpmGetIssuerHandles:1461 sshpmGetIssuerHandles: <issuer> L=San Jose, ST=California, C=US, O=Cisco Systems[TIME]: * sshpmGetIssuerHandles:1471 sshpmGetIssuerHandles: Mac Address in subject is 00:12:80:ad:7a:9c[TIME]: * sshpmGetIssuerHandles:1508 sshpmGetIssuerHandles: Cert is issued by Cisco Systems.[TIME]: * sshpmSsUserCertVerify:1212 ssphmSsUserCertVerify: self-signed user cert verfied.[TIME]: * sshpmGetIssuerHandles:1667 sshpmGetIssuerHandles: getting cisco ID cert handle...<SNIP> Self-Signed Certificate Check </SNIP>[TIME]: * sshpmGetCID:1932 sshpmGetCID: failed to find matching cert.[TIME]: * sshpmGetIssuerHandles:1775 sshpmGetIssuerHandles: SSC Key Hash is 98936bf9c90b30bf3c6bb9a0d7b23668887af49a* <SNIP> DEBU CTRLR </SNIP>
The Fix:(WLC_CLI) >config auth-list add ssc 00:12:80:ad:7a:9c 98936bf9c90b30bf3c6bb9a0d7b23668887af49a
In the WLC GUI, Go to: Security | AAA AP Policies
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 44BRKAGG-301114511_04_2008_c1
Does Regulatory Domain Matter? Yes!(WLC_CLI) >debug mac addr 00:12:80:ad:7a:9c(WLC_CLI) >debug lwapp events enable[TIME]: * spamVerifyRegDomain:6202 AP 00:12:80:ad:7a:9c
80211bg Regulatory Domain (-A) does not match with country (JP)reg. domain -JP for slot 0
[TIME]: DEBU CTRLR spamVerifyRegDomain:6167 spamVerifyRegDomain RegDomain set for slot 1 code 0 regstring -A regDfromCb -J
[TIME]: * spamVerifyRegDomain:6202 AP 00:12:80:ad:7a:9c 80211a Regulatory Domain (-A) does not match with country (JP) reg. domain -JP for slot 1
[TIME]: DEBU CTRLR spamVerifyRegDomain:6210 spamVerifyRegDomain AP RegDomain check for the country JP failed
[TIME]: * spamProcessConfigRequest:1730 AP 00:12:80:ad:7a:9c: Regulatory Domain check Completely FAILED. The AP will not be allowed to join.
The fix?Make sure you match your APs’ regulatory domain with your WLCs’.
How do you know how to make sure you do?Search CCO for “Wireless LAN Compliance Status”
Note: In the US, Your APs’ Regulatory Coding Is ‘– A’, not ‘– N’!!!
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
23
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 45BRKAGG-301114511_04_2008_c1
AP Join Problem – Path MTU/Firewall
X
1
43 2
4’
1. AP sends Discover packet (small) – gets thru
2. WLC sends Discover response (small) – gets thru
3. AP sends Join packet (small) – gets thru
4. WLC sends Join response (BIG) – first fragment gets thru, but
4’. The second fragment is blocked by the firewall
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 46BRKAGG-301114511_04_2008_c1
When Does AP Fail Over? What Happens Then?
APs will failover to other WLCs if the LWAPP control plane is interrupted
After either:
A missed heartbeat to WLC (sent every 30 seconds)
Or
A Non-ACK’d LWAPP control packet
Then:
The AP will send five successive heartbeats (each a second apart)
If no reply is received, the AP/WLC path is assumed down and the AP will attempt to join another controller
Likely causes:Wired network transmission problem
AP/WLC bug (key rotation problem, buffer leak)
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
24
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 47BRKAGG-301114511_04_2008_c1
Get Your APs to Join
So, make sure ALL WLCs in the cluster will properly allow all APs to join
Make sure all WLCs run the same software version
Make sure all WLCs are set to the correct time
Make sure all WLCs have all upgraded APs’ SSC hashes
See cisco.com Document ID: 99948, “Troubleshoot a Lightweight Access Point Not Joining a Wireless LAN Controller”
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 48BRKAGG-301114511_04_2008_c1
Q and A
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
25
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 49BRKAGG-301114511_04_2008_c1
Make Sure Stuff Basically Works
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 50BRKAGG-301114511_04_2008_c1
Make Sure It Basically Works
Turn it on and see if anyone complains
Walk around with a PC running a continuous ping to the default gateway
Walk around with a 7921 in a call to a 7960?Push the ? button the 7920 twice—monitor the RTP stats in real time
Set your PC’s interface MTU to the max, and download giant files
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
26
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 51BRKAGG-301114511_04_2008_c1
Get Trouble Spots Fixed Up
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 52BRKAGG-301114511_04_2008_c1
Fix up Trouble Spots
Reproduce problem at the application layer
Look at the area with a site survey type tool
Check for interference (SpEx)
Eliminate interference sources
Hand-tune RRM
Change or reorient antennas
Move or add APs
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
27
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 53BRKAGG-301114511_04_2008_c1
My Poorman’s Method
laptop with CB21AG, free Cisco Aironet Site Survey Utility, windump installed
have a continuous ping –t running to my default gateway
write all packets to/from the wireless adapter to a file:c:\>windump –p –i 3 –w d:\tmp\windump.enc
(interface # comes from windump –D)
In CASSU, start AP scan logging (log updates every five seconds)
now, walk around and make a note of where I am at times of high latency/packet loss
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 54BRKAGG-301114511_04_2008_c1
Poorman’s Method—CASSU
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
28
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 55BRKAGG-301114511_04_2008_c1
Poorman’s Method—Ping
Now, look in the packet capture (filtering on ICMP), and find the corresponding high latency times
74 ms latency!
74 ms latency!
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 56BRKAGG-301114511_04_2008_c1
Poorman’s Method—CASSU Scan Log
Now, look in the CASSU scan log (SST_APScanLog.txt) to see what the APs looked like, at the time of concern (17:50:28)
I can ignore all the BSSIDs except the ones advertising my SSID of interest
Hm ... no APs above -85dBm—I bet VoIP won’t work too well here ...
2008-04-27 17:50:29 ,blizzard,00:19:A9:41:AC:B0,-86,Secure,G,11 (2462),54,tuc-00-ap3,0,4,"CAC, CEC, QBSS, WMM, WPA, RM-Normal"2008-04-27 17:50:29 ,blizzard,00:19:A9:41:13:90,-85,Secure,G,11 (2462),54,tuc-00-ap1,0,4,"CAC, CEC, QBSS, WMM, WPA, RM-Normal"2008-04-27 17:50:29 ,blizzard,00:19:A9:41:0B:F0,-85,Secure,G,1 (2412),54,tuc-00-ap2,0,4,"CAC, CEC, QBSS, WMM, WPA, RM-Normal"
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
29
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 57BRKAGG-301114511_04_2008_c1
Q and A
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 58BRKAGG-301114511_04_2008_c1
Fix Individual Clients
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
30
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 59BRKAGG-301114511_04_2008_c1
Hand to Hand Combat with Clients
Client can’t talk to the networkAnatomy of a ping
“Autopsy tools”
“Random” client disconnects
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 60BRKAGG-301114511_04_2008_c1
DHCP
What it takes for your wireless client to ping (assuming EAP and DHCP, with L3 mobility configured)
Client probes for the SSIDClient authenticates/associates in 802.11 to an APEAP does its thingDHCP succeedsClient reaches RUN stateIP packet successfully transmitted by client over the air to AP, tunneled in LWAPP to WLC, (forwarded in EoIP to anchor WLC), decapsulatedby WLC to the wired network, forwarded to IP peer, and back again
IP
WLC
WLC
IP
IP
ACSLWAPP EO
IP
802.11 data
802.11 data
802.11 management
LWAPP802.11 management
RADIUSsupp.
driver
radio
EAPchan. 1
chan. 11
Anatomy of a Ping
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
31
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 61BRKAGG-301114511_04_2008_c1
What it takes for me to figure out why your wireless client can’t ping!
IP
WLC
WLC
IP
IP
ACSLWAPP EO
IP
802.11 data
802.11 data
802.11 management
LWAPP802.11 management
RADIUS
supp.driver
radio
EAP
Supplicantlogs
Driverdebugs/ adapter capture
chan. 1
chan. 11
wirelesssniff
wirelesssniff
APdebugs
wiredsniff
WLCdebugs
wiredsniff
ACSlogs
DHCP
DHCPlogs
NTP
Spectrum analysis
Autopsy of a Ping
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 62BRKAGG-301114511_04_2008_c1
<digression> Autopsy tools
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
32
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 63BRKAGG-301114511_04_2008_c1
Supplicant logs are needed to figure out what the client EAP supplicant is thinking. How to turn them on, and what they say, is entirely supplicant-specific.
WZC supplicant log:netsh ras set tracing * enabled —logs in c:\windows\tracingsee http://www.microsoft.com/technet/network/wifi/wlansupp.mspx
PROSet supplicant log: under hklm\software\intel\wireless\settings1xconfigdbg=wwxyz; 1xDebugLevel=dword:0x18;1xLogLevel=dword:0x18logs in c:\ (subject to change without warning)
ADU: see CSCsi16921CSSC: see Log Packager utility on cisco.com
IP
WLC
WLC
IP
IP
ACSLWAPP EO
IP
802.11 data
802.11 data
802.11 management
LWAPP802.11 management
RADIUSsupp.driver
radio
EAP
Supplicantlogs
chan. 1
chan. 11DHCP
Autopsy Tools—Supplicant
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 64BRKAGG-301114511_04_2008_c1
Driver debugs—use only under medical supervision (probably need a special driver build)Client adapter capture—capture a packet trace from the wireless adapter using Wireshark/windump in non-promiscuous mode (Windows)
Shows “Ethernet-II” packets at the NDIS layer (e.g. all IP packets sent to/from this client)Shows EAPOL packets (usually), so very helpful for EAP/supplicant troubleshootingDoes not show 802.11 management frames (no beacons, probes, authentication/association)
IP
WLC
WLC
IP
IP
ACSLWAPP EO
IP
802.11 data
802.11 data
802.11 management
LWAPP802.11 management
RADIUSsupp.driver
radio
EAPchan. 1
chan. 11DHCP
Driver debugs/adapter capture
Autopsy Tools—Client Driver
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
33
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 65BRKAGG-301114511_04_2008_c1
Wireless packet capture is essential for 802.11 support
Good options (Windows PCs):Omnipeek from Wildpackets (3945 with 10.5.1.75 driver or CB21AG)
Wireshark with CACE Technologies AirPcap adapters
USB adapters nice for multichannel sniff
AirMagnet
CommView for WiFi from Tamosoft
IP
WLC
WLC
IP
IP
ACSLWAPP EO
IP
802.11 data
802.11 data
802.11 management
LWAPP802.11 management
RADIUSsupp.driver
radio
EAPchan. 1
chan. 11DHCP
Wireless SniffWireless Sniff
Autopsy Tools—Wireless Sniff
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 66BRKAGG-301114511_04_2008_c1
Wireless Sniff—Some Tips
One separate packet capture per wireless channel of interest—a capture that scans across multiple channels is only useful for device discoveryAlways perform unfiltered captures unless you know there are no RF issues (need to see beacons, acks)Configure analyzer to cut a new file every 20–30MBConfigure analyzer not to display updated packet list during capture (reduce CPU load and minimize drops)When troubleshooting a roaming client, will need multiple analyzers moving in concert with the client under test (put everything on a cart and get some good exercise)To capture multiple channels at once, the CACE USB 11a/g adapter is a good option—put a bunch of them into the same USB hub and get it all on the same PCNTP sync everything
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
34
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 67BRKAGG-301114511_04_2008_c1
Use spectrum analysis to capture RF spectrum behavior—necessary to identify/track down non-802.11 interference sources
Cisco’s product: Spectrum Expert (nee Cognio)
IP
WLC
WLC
IP
IP
ACSLWAPP EO
IP
802.11 data
802.11 data
802.11 management
LWAPP802.11 management
RADIUSsupp.driver
radio
EAPchan. 1
chan. 11DHCP
Spectrum Analysis
Autopsy Tools—Spectrum Analysis
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 68BRKAGG-301114511_04_2008_c1
SpEx Spectrogram
Very Low Power/Activity
Microwave Oven
Active AP
Mouse Hover Reveals Channel Position,
Sweep Time/Date, and Top Five Devices Color Indicates Power,
Red = -35 dBm
High Power, Bursting over Time
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
35
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 69BRKAGG-301114511_04_2008_c1
SpEx Tips
When capturing, be sure to have an 802.11 adapter installed, enabled, but configured not to associate to a WLAN
Spectrum Expert cannot identify 802.11 devices (MAC address, etc.) without an 802.11 adapter’s aid
NTP sync your SpEx host!
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 70BRKAGG-301114511_04_2008_c1
To Collect Debugs from LWAPP IOS APs, You Can:
connect via console (use hidden debug lwapp console cli then conf t to set the console speed to 115200)
from WLC CLI, use:debug ap enable APnamedebug ap command “debug command” APname
in 5.0, can use telnet/ssh to connect to an LWAPP IOS AP
IP
WLC
WLC
IP
IP
ACSLWAPP EO
IP
802.11 data
802.11 data
802.11 management
LWAPP802.11 management
RADIUSsupp.driver
radio
EAPchan. 1
chan. 11DHCP
LWAPP (IOS) AP Debugs
Autopsy Tools—AP Debugs
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
36
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 71BRKAGG-301114511_04_2008_c1
AP Debugs—Tips
By default, radio debugs (debug dot11 dot11radiox) appear only on the console. To see radio debugs in your telnet/ssh/WLC CLI session, use the command no debug dot11 dot11radiox printf, where x is 0 or 1
Useful radio debugs:debug dot11 dot11radiox trace print mgmt keys beacon rcv xmt (beacon, rcv, xmt apt to be extremely verbose!)
Useful LWAPP join debugs:debug dhcpdebug ip udpdebug lwapp client {config, error, event, packet}
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 72BRKAGG-301114511_04_2008_c1
Autopsy Tools—Wired Sniff
When capturing from trunk ports, always capture with 802.1q tags(watch out for packets in the wrong VLANs)
Cut new file every 20/30MB; don’t display packet updates in real time
NTP sync your sniffers!
IP
WLC
WLC
IP
IP
ACSLWAPP EO
IP
802.11 data
802.11 data
802.11 management
LWAPP802.11 management
RADIUSsupp.driver
radio
EAP
Wired Capture
chan. 1
chan. 11DHCP
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
37
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 73BRKAGG-301114511_04_2008_c1
Autopsy Tools—WLC Debugs
Capture WLC debugs from a telnet/ssh or console (115200 bps) session
Simplest debug for one client under test:debug client MACaddress
enables basic dot11, dot1x, pem and dhcp debugs
IP
WLC
WLC
IP
IP
ACSLWAPP EO
IP
802.11 data
802.11 data
802.11 management
LWAPP802.11 management
RADIUSsupp.driver
radio
EAP
WLC debugs
chan. 1
chan. 11DHCP
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 74BRKAGG-301114511_04_2008_c1
WLC Debugs
More general client debugging options:debug dot11debug dot1xdebug aaa <= use for RADIUS troubleshootingdebug pemdebug mobility <= handoffsdebug dhcpUse debug mac MACaddr to filter on a single client
NTP sync your WLC!
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
38
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 75BRKAGG-301114511_04_2008_c1
Autopsy Tools—RADIUS Logs
See “Logs and Reports” section of the ACS User GuideSystem Configuration -> Service Control -> Full for a deep dive, but beware memory/disk exhaustion—use under medical supervisionNTP sync your ACS!
IP
WLC
WLC
IP
IP
ACSLWAPP EO
IP
802.11 data
802.11 data
802.11 management
LWAPP802.11 management
RADIUSsupp.driver
radio
EAP
RADIUS Logs
chan. 1
chan. 11DHCP
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 76BRKAGG-301114511_04_2008_c1
Autopsy Tools—DHCP Logs
IOS DHCP server:debug ip dhcp server eventsdebug ip dhcp server packet
IP
WLC
WLC
IP
IP
ACSLWAPP EO
IP
802.11 data
802.11 data
802.11 management
LWAPP802.11 management
RADIUSsupp.driver
radio
EAP
DHCP logs
chan. 1
chan. 11DHCP
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
39
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 77BRKAGG-301114511_04_2008_c1
Autopsy Tools </digression>
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 78BRKAGG-301114511_04_2008_c1
Client probes for the SSID
Client authenticates/associates in 802.11 to an AP
EAP does its thing
DHCP succeeds
Client reaches RUN state
IP
WLC
WLC
IP
IP
ACSLWAPP EO
IP
802.11 data
802.11 data
802.11 management
LWAPP802.11 management
RADIUSsupp.
driver
radio
EAPchan. 1
chan. 11
Probe Req
Probe ReqProbe Req
Probe Req
Probe Resp
DHCP
Anatomy of a Ping—Probing
Probe Resp
wireless sniffwireless sniff
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
40
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 79BRKAGG-301114511_04_2008_c1
Autopsy—Probing
Clients broadcasts a probe for the SSID of interest
AP hopefully unicasts back a probe response
Probe response includes interesting facts (Information Elements) about the service
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 80BRKAGG-301114511_04_2008_c1
Problems at Probing Stage
What if the client never sends out a probe?Is it configured for the SSID of interest?
What if the AP doesn’t send back the probe response?Is it (WLC) configured for the SSID of interest?
Do you have RF coverage from this AP? (can you see beacons from it?)
What if the client never moves beyond probing?Does it like the IEs that the AP is sending out? Try different crypto settings; disable Aironet extensions; try different basic rates; etc.
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
41
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 81BRKAGG-301114511_04_2008_c1
DHCP
Client probes for the SSID
Client authenticates/associates in 802.11 to an AP
EAP does its thing
DHCP succeeds
Client reaches RUN state
IP
WLC
WLC
IP
IP
ACSLWAPP EO
IP
802.11 data
802.11 data
802.11 management
LWAPP802.11 management
RADIUSsupp.
driver
radio
EAPchan. 1
chan. 11
Anatomy—802.11 Auth/Assoc
Wireless Sniff WLC Debugs
Association
AuthenticationAuthentication
Association
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 82BRKAGG-301114511_04_2008_c1
Autopsy—802.11 Auth/Assoc
Client and AP authenticate to each other (normally just Open authentication nowadays)
Client tries to associate to the AP, hopefully gets a status=0 (successful) response
What if unsuccessful?Check status code
Run debugs on WLC
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
42
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 83BRKAGG-301114511_04_2008_c1
Client probes for the SSID
Client authenticates/associates in 802.11 to an AP
EAP does its thing
DHCP succeeds
Client reaches RUN state
IPIP
WLC
WLC
IP ACSLWAPP EO
IP
802.11 data
802.11 data
802.11 management
LWAPP802.11 management
RADIUSsupp.
driver
radio
EAPchan. 1
chan. 11
EAPOL Start
EAP SUCCESS
EAP ID Response
EAP Request IDRADIUS Access-Request
EAP blah blah blah blah
DHCP
Anatomy—EAP Does Its Thing
Driver Debugs/Adapter CaptureSupplicant
LogsWLC
DebugsRADIUS
Logs
pass keys all around the place
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 84BRKAGG-301114511_04_2008_c1
802.11 Capture of 802.1X MS-PEAP (dWEP)
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
43
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 85BRKAGG-301114511_04_2008_c1
Wireshark Capture of MS-PEAP (WPA2)
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 86BRKAGG-301114511_04_2008_c1
Successful 802.1X Client Authentication
(WLC_CLI) >debug mac addr 00:13:ce:57:2b:84(WLC_CLI) >debug dot1x events enable [TIME]: * dot1x_auth_txReqId:2827 Sending EAP-Request/Identity to mobile 00:13:ce:57:2b:84 (EAP Id 1)[TIME]: * dot1x_authsm_capture_supp:675 Received EAPOL START from mobile 00:13:ce:57:2b:84[TIME]: * dot1x_handle_eapsupp:1962 Received Identity Response (count=n) from mobile 00:13:ce:57:2b:84<SNIP> Series of 802.1X EAP Requests/Responses </SNIP>[TIME]: * dot1x_process_aaa:898 Processing Access-Challenge for mobile 00:13:ce:57:2b:84[TIME]: * dot1x_bauthsm_txReq:465 Sending EAP Request from AAA to mobile 00:13:ce:57:2b:84 (EAP Id 19)[TIME]: * dot1x_handle_eapsupp:1997 Received EAP Response from mobile 00:13:ce:57:2b:84 (EAP Id 19, EAP
Type 25)[TIME]: * dot1x_process_aaa:906 Processing Access-Accept for mobile 00:13:ce:57:2b:84[TIME]: * createNewPmkCacheEntry:691 Creating a new PMK Cache Entry for station 00:13:ce:57:2b:84 (RSN 0)[TIME]: * dot1x_auth_txCannedSuccess:2594 Sending EAP-Success to mobile 00:13:ce:57:2b:84 (EAP Id 19)[TIME]: * sendDefaultRc4Key:450 Sending default RC4 key to mobile 00:13:ce:57:2b:84[TIME]: * sendKeyMappingRc4Key:325 Sending Key-Mapping RC4 key to mobile 00:13:ce:57:2b:84[TIME]: * dot1x_trans_authsm:2448 Received Auth Success while in Authenticating state for mobile
00:13:ce:57:2b:84
* <SNIP> DEBU STA 00:13:ce:57:2b:84 </SNIP>
debug dot1x events
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
44
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 87BRKAGG-301114511_04_2008_c1
Failed 802.1X Client Authentication
(WLC_CLI) >debug mac addr 00:13:ce:57:2b:84(WLC_CLI) >debug dot1x events enable [TIME]: * dot1x_auth_txReqId:2827 Sending EAP-Request/Identity to mobile 00:13:ce:57:2b:84
(EAP Id 1)[TIME]: * dot1x_authsm_capture_supp:675 Received EAPOL START from mobile
00:13:ce:57:2b:84[TIME]: * dot1x_handle_eapsupp:1962 Received Identity Response (count=n) from mobile
00:13:ce:57:2b:84<SNIP> Series of 802.1X EAP Requests/Responses </SNIP>[TIME]: * dot1x_process_aaa:898 Processing Access-Challenge for mobile 00:13:ce:57:2b:84[TIME]: * dot1x_bauthsm_txReq:465 Sending EAP Request from AAA to mobile 00:13:ce:57:2b:84
(EAP Id 14)[TIME]: * dot1x_handle_eapsupp:1997 Received EAP Response from mobile 00:13:ce:57:2b:84
(EAP Id 14, EAP Type 25)[TIME]: * dot1x_process_aaa:928 Processing Access-Reject for mobile 00:13:ce:57:2b:84[TIME]: * dot1x_auth_txCannedFail:2865 Sending EAP-Failure to mobile 00:13:ce:57:2b:84 (EAP
Id 14)
* <SNIP> DEBU STA 00:13:ce:57:2b:84 </SNIP>
debug dot1x events—Username/Password Failure
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 88BRKAGG-301114511_04_2008_c1
Check Client Record for Details
In the WLC GUI, Go to: Wireless | Clients and Select Details for the Client of Choice
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
45
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 89BRKAGG-301114511_04_2008_c1
Successful 802.1X Client Authentication
(WLC_CLI) >debug mac addr 00:13:ce:57:2b:84(WLC_CLI) >debug aaa events enable [TIME]: * sendRadiusMessage:2494 Successful transmission of Authentication Packet (id 49) to
20.20.20.12:1812, proxy state 00:13:ce:57:2b:84-ce:57[TIME]: DEBU CTRLR processIncomingMessages:3480 ****Enter processIncomingMessages: response
code=11[TIME]: DEBU CTRLR processRadiusResponse:3053 ****Enter processRadiusResponse: response
code=11[TIME]: * processRadiusResponse:3325 Access-Challenge received from RADIUS server 20.20.20.12 for
mobile 00:13:ce:57:2b:84 receiveId = 2[TIME]: * sendRadiusMessage:2494 Successful transmission of Authentication Packet (id 59) to
20.20.20.12:1812, proxy state 00:13:ce:57:2b:84-ce:57[TIME]: DEBU CTRLR processIncomingMessages:3480 ****Enter processIncomingMessages: response
code=2[TIME]: DEBU CTRLR processRadiusResponse:3053 ****Enter processRadiusResponse: response code=2[TIME]: * processRadiusResponse:3325 Access-Accept received from RADIUS server 20.20.20.12 for
mobile 00:13:ce:57:2b:84 receiveId = 2
* <SNIP> DEBU STA 00:13:ce:57:2b:84 </SNIP>
debug aaa events
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 90BRKAGG-301114511_04_2008_c1
Failed 802.1X Client Authentication
AAA connectivity failure will generate an SNMP trap
debug aaa events—AAA Server Unreachable
(Cisco Controller) >debug mac addr 00:13:ce:57:2b:84(Cisco Controller) >debug aaa events enable [TIME]: * sendRadiusMessage:2494 Successful transmission of Authentication Packet (id 66) to 20.20.20.12:1812, proxy state 00:13:ce:57:2b:84-ce:57[TIME]: * sendRadiusMessage:2494 Successful transmission of Authentication Packet (id 66) to 20.20.20.12:1812, proxy state 00:13:ce:57:2b:84-ce:57[TIME]: * sendRadiusMessage:2494 Successful transmission of Authentication Packet (id 66) to 20.20.20.12:1812, proxy state 00:13:ce:57:2b:84-ce:57[TIME]: * sendRadiusMessage:2494 Successful transmission of Authentication Packet (id 66) to 20.20.20.12:1812, proxy state 00:13:ce:57:2b:84-ce:57[TIME]: * sendRadiusMessage:2494 Successful transmission of Authentication Packet (id 66) to 20.20.20.12:1812, proxy state 00:13:ce:57:2b:84-ce:57[TIME]: * sendRadiusMessage:2494 Successful transmission of Authentication Packet (id 66) to 20.20.20.12:1812, proxy state 00:13:ce:57:2b:84-ce:57[TIME]: * radiusProcessQueue:2735 Max retransmission of Access-Request (id 66) to 20.20.20.12 reached for mobile 00:13:ce:57:2b:84[TIME]: * sendAAAError:323 Returning AAA Error 'Timeout' (-5) for mobile 00:13:ce:57:2b:84
* <SNIP> DEBU STA 00:13:ce:57:2b:84 </SNIP>
In the WLC GUI, Go to: Management | SNMP Trap Logs
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
46
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 91BRKAGG-301114511_04_2008_c1
Verify Complete 802.11/802.1X Connectivity
(WLC_CLI) >debug mac addr 00:13:ce:57:2b:84(WLC_CLI) >debug pem state enable[TIME]: pem_api.c:1780 - State Update 00:13:ce:57:2b:84 from RUN (20) to START (0)[TIME]: pem_api.c:1836 - State Update 00:13:ce:57:2b:84 from START (0) to AUTHCHECK (2)[TIME]: pem_api.c:1859 - State Update 00:13:ce:57:2b:84 from AUTHCHECK (2) to 8021X_REQD (3)[TIME]: pem_api.c:3977 - State Update 00:13:ce:57:2b:84 from 8021X_REQD (3) to L2AUTHCOMPLETE (4)[TIME]: pem_api.c:4152 - State Update 00:13:ce:57:2b:84 from L2AUTHCOMPLETE (4) to RUN (20)
debug pem state
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 92BRKAGG-301114511_04_2008_c1
Troubleshooting 802.1X
Make sure the RADIUS server is properly configured
Make Sure the Correct Shared Secret Is Input
Select the Correct RADIUS Port (Common Ports Are 1812 and 1645)
Status Must Be Enabled
Network User Auth Has to Be Enabled for This AAA Server to Be Used
In the WLC GUI, Go to: Security | AAA RADIUS Authentication and Then Select Edit or New
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
47
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 93BRKAGG-301114511_04_2008_c1
Troubleshooting 802.1X
Make sure the proper security policy is enabled for both encryption and authentication
Step (1): Select the Desired Layer 2 Security Configuration
Step (2): Ensure at Least One RADIUS Server Is Configured per WLAN. Configure Additional Ones for Fedundancy
In the WLC GUI, Go to: WLANs | WLANs WLANs and Then Select Edit for the WLAN of Interest
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 94BRKAGG-301114511_04_2008_c1
Troubleshooting 802.1X
Enabled Logging in your ACS server to identify where issues might lie with backend authentication
Make Sure at Least Logging for Failed Attempts Is Enabled on ACS So Server-side Debugging Can Be Performed
In ACS, Select System Configuration | Logging and Enable Each Desired Option
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
48
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 95BRKAGG-301114511_04_2008_c1
IP
Anatomy—DHCP Succeeds
Client probes for the SSIDClient authenticates/associates in 802.11 to an APEAP does its thingDHCP succeedsClient reaches RUN state
IP
WLC
WLC
IP ACSLWAPP EO
IP
802.11 data
802.11 data
802.11 management
LWAPP802.11 management
RADIUSsupp.
driver
radio
EAPchan. 1
chan. 11DHCP
DHCP discover
DHCP offerDHCP discover
DHCP offer
DHCP requestDHCP request
DHCP ack
DHCP ack
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 96BRKAGG-301114511_04_2008_c1
Client IP Provisioning via DHCP
(WLC_CLI) >debug mac addr 00:13:ce:57:2b:84(WLC_CLI) >debug dhcp message enable [TIME]: dhcp option: received DHCP DISCOVER msg<SNIP> DHCP Discover message details </SNIP>[TIME]: Forwarding DHCP packet (332 octets) from 00:13:ce:57:2b:84
-- packet received on direct-connect port requires forwarding to external DHCP server. Next-hop is 20.20.20.1
[TIME]: dhcp option: received DHCP OFFER msg[TIME]: dhcp option: server id = 20.20.20.1[TIME]: dhcp option: netmask = 255.255.255.0[TIME]: dhcp option: gateway = 20.20.20.1<SNIP> DHCP Offer message details </SNIP>[TIME]: dhcp option: received DHCP REQUEST msg[TIME]: dhcp option: requested ip = 20.20.20.113[TIME]: dhcp option: server id = 1.1.1.1<SNIP> DHCP Request message details </SNIP>[TIME]: Forwarding DHCP packet (340 octets) from 00:13:ce:57:2b:84
-- packet received on direct-connect port requires forwarding to external DHCP server. Next-hop is 20.20.20.1
[TIME]: dhcp option: received DHCP ACK msg<SNIP> DHCP Ack message details </SNIP>
debug dhcp message
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
49
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 97BRKAGG-301114511_04_2008_c1
Troubleshooting DHCP
Clients are not configured for static addressing
DHCP scopes are properly configured (either external or internal DHCP)
External servers: Need to support DHCP proxy—if they don’t, turn on bridging:(WLC_CLI) >config dhcp proxy disable
Internal DHCP server: After properly configuring the WLC’s scopes, each interface needs to have the WLC’s management IP as its DHCP server IP address, as below:
In the WLC GUI, Go to: Controller | Interfaces and
Select Edit for the Interface of Choice
For Internal DHCP, Input the WLC’s Management IP Address Here
Note: The WLC’s Internal DHCP Server Will Provide Addresses to APs, As Well, Provided the WLC Is Running 4.0 or Later and the AP DHCP Requests Can Find the Controller’s Management Interface
If Clients Aren’t Getting Addresses Properly via DHCP, Ensure:
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 98BRKAGG-301114511_04_2008_c1
Anatomy—PING Succeeds!!
Client probes for the SSID
Client authenticates/associates in 802.11 to an AP
EAP does its thing
DHCP succeeds
Client reaches RUN state
IP
WLC
WLC
IP
IP
ACSLWAPP EO
IP
802.11 data
802.11 data
802.11 management
LWAPP802.11 management
RADIUSsupp.driver
radio
EAPchan. 1
chan. 11DHCP
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
50
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 99BRKAGG-301114511_04_2008_c1
ACS
DHCP
Now for the Hard Part—Troubleshooting Roaming
Take this ...
And move the client under test through it, inthree dimensions, in real time!
IP
WLC
WLC
IP
IP
LWAPP EOIP
802.11 data
802.11 data
802.11 management
LWAPP802.11 management
RADIUSsupp.driverradio
EAPchan. 1
chan. 11
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 100BRKAGG-301114511_04_2008_c1
Troubleshooting a Roaming Client in SituIs Very Hard—You Don’t Want to Do This
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
51
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 101BRKAGG-301114511_04_2008_c1
So Instead, Factor Out Variables
Does the roaming problem happen when using open auth/no crypto? (Factor out: EAP, CCKM, PMK caching, etc.)
Does the roaming problem with intracontroller roams, or only intercontroller? (Factor out: VLAN configproblems, CAM table, L3 mobility problems, mobility group config problems)
Does the roaming problem occur only in specific locations? (Factor out: RF coverage issues)
Does the problem happen when using one supplicant, but not another? (Factor out: specific supplicant issues)
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 102BRKAGG-301114511_04_2008_c1
Do Some Clients Have Roaming Problems, and Others Not?
Try tuning the client roaming behaviorIntel: Roaming Aggressiveness knob
7921: lock to 802.11a if you have the coverage
CB21ag: turn down Scan Valid, BSS Aging in Device Manager (see “Optimize CB21AG/PI21AG Roaming Behavior”, Document ID 69403, cisco.com)
Try upgrading the client code7921: must have at least 1.0.5
Intel: drivers in latest (April ’08) 11.5.1.2 bundle
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
52
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 103BRKAGG-301114511_04_2008_c1
What If Some Clients Just Don’t Roam Right, No Matter What?
Prove that another wireless adapter (CB21AG?) in the identical application, works fine
Escalate to your laptop/device vendor
Open a case with Cisco, if TAC assistance is needed in setting up the back end debugging
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 104BRKAGG-301114511_04_2008_c1
Q and A
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
53
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 105BRKAGG-301114511_04_2008_c1
Recommended Reading
802.11 Wireless Networks, 2nd Ed., Matthew Gast, O’ReillyReal 802.11 Security, Edney and Arbaugh, Addison-Wesley“Radio Resource Management under Unified Wireless Networks”, Document ID 71113, cisco.com“Troubleshoot a Lightweight Access Point Not Joining a Wireless LAN Controller”, Document ID: 99948, cisco.com
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 106BRKAGG-301114511_04_2008_c1
Complete Your Online Session Evaluation
Give us your feedback and you could win fabulous prizes. Winners announced daily.
Receive 20 Passport points for each session evaluation you complete.
Complete your session evaluation online now (open a browser through our wireless network to access our portal) or visit one of the Internet stations throughout the Convention Center.
Don’t forget to activate your Cisco Live virtual account for access to all session material on-demand and return for our live virtual event in October 2008.
Go to the Collaboration Zone in World of Solutions or visit www.cisco-live.com.
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
54
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 107BRKAGG-301114511_04_2008_c1