Blackhole traffic issue related with static EtherChannel (Mode on) and how LACP can prevent blackhole traffic issue

Blackhole issue on  static  port-channel due to one of link in port-channel is down

The network draw and static port-channel configuration are below. use cisco 3750 as the example. 

Static layer2 Etherchannel configuration on switch A

interface Port-channel1

 switchport trunk encapsulation dot1q

 switchport trunk allowed vlan 100,200,300,301

 switchport mode trunk

interface FastEthernet1/0/1

 switchport trunk encapsulation dot1q

 switchport trunk allowed vlan 100,200,300,301

 switchport mode trunk

 speed 100

 duplex full

 channel-group 1 mode on

interface FastEthernet1/0/2

 switchport trunk encapsulation dot1q

 switchport trunk allowed vlan 100,200,300,301

 switchport mode trunk

 speed 100

 duplex full

 channel-group 1 mode on

Static layer2 Etherchannel configuration on switch B

interface Port-channel1

 switchport trunk encapsulation dot1q

 switchport trunk allowed vlan 100,200,300,301

 switchport mode trunk

interface FastEthernet1/0/3

 switchport trunk encapsulation dot1q

 switchport trunk allowed vlan 100,200,300,301

 switchport mode trunk

 speed 100

 duplex full

 channel-group 1 mode on

interface FastEthernet1/0/4

 switchport trunk encapsulation dot1q

 switchport trunk allowed vlan 100,200,300,301

 switchport mode trunk

 speed 100

 duplex full

 channel-group 1 mode on

The below draw illustrate the blackhole traffic issue due to one of link down in static port-channel. use cisco 3750 as the example

If the link A1-B1 is down, but interface F1/0/1 on switch A or interface F1/0/3 on switch B stay up. the blackhole traffic will happen. 

Let's say interface F1/0/1 on switch A stay up but the link A1-B1 is down .  Interface F1/0/1 of switch A will still stay in port-channel due to static port-channel configuration. The traffic which is from switch A towards to switch B is sent to interface port-channel1, then some of traffic will be distributed into interface F1/0/1 of switch A since interface F1/0/1 is still bundled in port-channel1. since link A1-B1 is down,These traffic can't go out from interface F1/0/1 of  switch A, so these traffic will be dropped. This is the blackhole traffic issue.

The symptom of this blackhole traffic issue is some hosts on switch B is reachable from switch A, but others is not.

There is command to determine which interface in the EtherChannel forwards specified traffic, command is test etherchannel load-balance interface port-channel number {ip | l4port | mac} [source_ip_add | source_mac_add | source_l4_port] [dest_ip_add | dest_mac_add | dest_l4_port]. 

you need know what load-balance method is using for port-channel, then use the test etherchannel load-balance interface port-channel number command accordingly.  for example, if the load-balance is src-mac, then you can issue test etherchannel load-balance interface port-channel number  mac  source_mac_add command

Blackhole issue on  static  port-channel due to VLAN miss match on one of physical port

If someone add vlan on interface F1/0/1 of switch A directly, the blackhole traffic issue will happen.  

The command of adding vlan on interface F1/0/1 of switch A is below:

! On switch A

interface F1/0/1
switchport trunk allowed vlan add 500


The reason for blackhole traffic issue is below:

Since it add one more Vlan on interface F1/0/1,  the vlan member on interface F1/0/1 will be different with vlan membe in port-channel interface. F1/0/1 is suspend due to the VLAN member on interface F1/0/1 miss match with VLAN member on port-channel interface. F1/0/1 is blocked. however F1/0/3 which is related peer port on switch B know nothing about F1/0/1 is suspend due to the static port-channel configuration. So F1/0/3 will still stay in port-channel.  Traffic from switch B to switch A will go to port-channel, then those traffic will be distributed to interface F1/0/3 and F1/0/4 on switch B. Traffic will be dropped silently after it be distributed onto interface F1/0/3. This is the reason of blackhole traffic issue.

LACP and PAgP both can prevent the blackhole issue

We use below LACP port-channel configuration and diagram to illustrate how LACP port-channel prevent the blackhole issue. the switch is cisco3750

SwitchA configuration

interface Port-channel1
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 100,200,300,301
 switchport mode trunk

interface FastEthernet1/0/1
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 100,200,300,301
 switchport mode trunk
 speed 100
 duplex full
 channel-group 1 mode active

interface FastEthernet1/0/2
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 100,200,300,301
 switchport mode trunk
 speed 100
 duplex full
 channel-group 1 mode active


SwitchB configuration

interface Port-channel1
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 100,200,300,301
 switchport mode trunk

interface FastEthernet1/0/3
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 100,200,300,301
 switchport mode trunk
 speed 100
 duplex full
 channel-group 1 mode active

interface FastEthernet1/0/4
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 100,200,300,301
 switchport mode trunk
 speed 100
 duplex full
 channel-group 1 mode active

LACP prevent the blackhole issue from one of link in port-channel is down

If the link A1-B1 down,  however interface F1/0/1 and F1/0/3 both stay up. In this situation LACP can't receive LACP packet from each other over the link A1-B1, so LACP put F1/0/1 on switch A and F1/0/3 on switch B to stand-alone mode, which means interface F1/0/1 and interface F1/0/3 is no longer in the port-channel, it just regular switch port now.   since  link A1-B1  is down, so no mac address will be learned on F1/0/1 and F1/0/3, therefore no traffic will be forward on F1/0/1 of switch A and F1/0/3 of switch B.  so there is no blackhole traffic happen.

LACP prevent the blackhole issue from VLAN miss match on one of physical port

Let's say vlan 50 is added on physical port F1/0/1 on switch A, then F1/0/1 is suspend due to vlan in F1/0/1 mismatch with interface port-channel 1. in suspend stage, the F1/0/1 only receive LACP PDU, but not send LACP PDU. it also block Data traffic. since F1/0/1 is out of port-channel bundle, so no port-channel traffic will split into F1/0/1;
F1/0/3 on switch B first will change to waiting stage for less than one minute, then change to stand-alone mode since it doesn't receive LACP PDU. since F1/0/3 is out of port-channel bundle now, so no port-channel traffic will split in to F1/0/3.  so there is no blackhole traffic issue. also since the peer port F1/0/1 on switch A block Data traffic, so there is no second data connection between switch A and switch B, so there is no loop issue.

Below is adding vlan 50 and port stage changes

switchA(config)#interface FastEthernet1/0/1
switchA(config-if)#switchport trunk allowed vlan add 50


switchA#sh eth su
Flags:  D - down        P - bundled in port-channel
        I - stand-alone s - suspended
        H - Hot-standby (LACP only)
        R - Layer3      S - Layer2
        U - in use      f - failed to allocate aggregator

        M - not in use, minimum links not met
        u - unsuitable for bundling
        w - waiting to be aggregated
        d - default port


Number of channel-groups in use: 1
Number of aggregators:           1

Group  Port-channel  Protocol    Ports
------+-------------+-----------+-----------------------------------------------
1      Po1(SU)         LACP      Fa1/0/1(s)  Fa1/0/2(P)  

F1/0/3 in waiting stage for around 30 seconds depend on LACP times

switchB#sh eth su
Flags:  D - down        P - bundled in port-channel
        I - stand-alone s - suspended
        H - Hot-standby (LACP only)
        R - Layer3      S - Layer2
        U - in use      f - failed to allocate aggregator

        M - not in use, minimum links not met
        u - unsuitable for bundling
        w - waiting to be aggregated
        d - default port


Number of channel-groups in use: 1
Number of aggregators:           1

Group  Port-channel  Protocol    Ports
------+-------------+-----------+-----------------------------------------------
1      Po1(SU)         LACP       Fa1/0/3(w) Fa1/0/4(P)


Final port stage on switch B

switchB#
switchB#sh eth su
Flags:  D - down        P - bundled in port-channel
        I - stand-alone s - suspended
        H - Hot-standby (LACP only)
        R - Layer3      S - Layer2
        U - in use      f - failed to allocate aggregator

        M - not in use, minimum links not met
        u - unsuitable for bundling
        w - waiting to be aggregated
        d - default port


Number of channel-groups in use: 1
Number of aggregators:           1

Group  Port-channel  Protocol    Ports
------+-------------+-----------+-----------------------------------------------
1      Po1(SU)         LACP      Fa1/0/3(I)  Fa1/0/4(P)

LACP prevent the one-way loop from unidirectional link

 If the problem of link A1-B1 is unidirectional issue which means link A1-B1 pass traffic from switch A to switch B, but it doesn't pass traffic from switch B to switch A. also F1/0/1 on switch A and F1/0/3 on switchB both stay up. How can LACP prevent this one-way loop?

The below is the explanation of how LACP prevent the one-way loop issue:

Since link A1-B1 can pass the traffic from switch A to switch B,  so port F1/0/3 on switch B can receive LACP PDU, but it can't further negotiate the LACP PDU with interface F1/0/1of switch A due to no LACP PDU can pass from switch B to switch A. so interface F1/0/3 on switch B LACP status oscillate between waiting stage and Hot-standby. Since Data traffic is blocked in either waiting stage or Hot-standby.  therefore one-way loop can't happen.  

Since F1/0/1 on switch A can't receive LACP PDU due to link A1-B1 has unidirectional issue. so F1/0/1 will be not bundled in port-channel and put into stand-alone stage.  even though F1/0/1 on switch A is in stand-alone stage, its peer port F1/0/3 on switch B is in LACP waiting stage or hot-standy stage which Data traffic is blocked in either stage, so it prevent the loop. 

For Network Design, Build and Consulting Services, 

please visit:

www.unistreamnetworks.com