Tonight I faced a DoS attack, first sign is all VLAN has unstable connectivity the packet loss ratio was very high when I pinged from a PC in this VLAN, then other VLAN was impacted randomly.
The unstable VLAN was across in two switches in the access layer and the collapse core layer, in both access layer the packet loss ratio is very high when I pinged the gateway interface of this VLAN (Its SVI, the interface VLAN in the core switch).
Checking the CPU usage of all switches in the access layer and core switches I notice that just only in the switches were the unstable VLAN is standing, they are SWC, SWA1 and SWA2 are facing a very high CPU usage 99%, the rest of the switches has normal usage “< 20%”
(All swiches are Cisco Catalyst 4500)
Isolating SWA1 and SWA4, juts shut down their uplinks, the CPU usage of SWC went down from 100% to 20%, at the same time SWA4’s CPU usage went down under 20% as well, the SWA1’s CPU usage kept its ratio at 99%, cheking SWA1’s CPU processes I got the same result.
Considering that the process “Cat4K Mgmt LoPri” has almost all CPU utilization (95,59%), which means that background and low-priority processes are the troublemaker. Either “Cat4k Mgmt HiPri” and “Cat4k Mgmt LoPri” aggregate multiple platform-specific processes essential for management functiont on the Catalyt 4500. Next step is to see the platform-specific processes use the CPU under the context these two processes (HiPri and LoPri).
The platform-process “K5L3Unicast Adj Tabl”, it is consumes high CPU usage. This platform-process is active when a new MAC address has been learned and the adjacency table is rewritten. This takes place when in the switch receives an unknown source MAC address, it’s forwarded to the CPU for MAC address learning.
It could be possible to have a device connected to a switch port sending lots of MAC address randomly and massively forcing the switch to spend all CPU capacity just to process the new MAC addresses.
So next step is check the interface statistics looking for the highest traffic rate, and the two highest are the following:
Considering the GigabitEthernet4/2 traffic is 29 times higher than GigabitEthernet3/42 I shut down the interface GigabitEthernet4/2 and after this, the CPU utilization recover a normal rate, as we can see below:
Finally we recognize the device connected to this switch port, it was a “time capsule” which is a harddisc which is shared in the network with Ethernet ports (4 ports) and with Wi-Fi, the device was disconnected and now I am going to investigate what application into this device was executing the DoS attack.
To be fair, here the source I considered to write this article:
- Cisco System, “High CPU Utilization on Cisco IOS Software-Based Catalyst 4500 Switches”.
http://www.cisco.com/en/US/products/hw/switches/ps663/products_tech_note09186a00804cef15.shtml - LearnIOS, “CPU99%”.
http://www.learnios.com/viewtopic.php?f=16&t=13675&start=5
HenryTips, network 
#1 by DAN on 2012/09/12 - 5:56 pm
Spot on buddy!! Thank you so much for this excellent post! :@)
#2 by Grace on 2013/01/29 - 12:23 pm
There are actually lots of details like that to take into consideration.
That is a nice point to carry up. I provide the ideas above as
general inspiration but clearly there are questions just like the one you deliver up where
a very powerful factor can be working in sincere good faith.
I don?t know if finest practices have emerged round things like that,
however I’m certain that your job is clearly identified as a fair game. Both boys and girls feel the impression of just a second’s pleasure, for the remainder of their lives.