Cisco ACI APIC – M3/L3 – fabric discovery

ACI APIC - M3/L3 Fabric Discovery

We have completed a number of Cisco ACI installations over the last few months and with the EoS/EoL announcement of the APIC-L2 and APIC-M2 models, we felt that we should share our experience.

“APIC-M3 and APIC-L3"

When ordering a Cisco ACI Fabric now, you should expect to have them shipped with either APIC-L3 or the APIC-M3 clusters (depending on your fabric size). The new APIC models ship with new internal hardware. The most apparent and potentially impacting change is the Virtual Interface Card (VIC).

The APIC-L3 and APIC-M3 now ship with the newer Cisco 1455 VIC card, which provides 4 SFP interfaces of 10/25-Gbps, compared to the older APIC-L2 and APIC-M2 appliances that ship with only 2 SFP interfaces of 10-Gbps.

This is where a change in behaviour was observed. With the new VIC cards, the physical fabric connectivity needs to change. For example, how the APICs physically attach to the leaf switches.

“Fabric discovery issues”

The new recommended best-practice deployment is to connect ports 1 and 3 on the VIC of each APIC to different leaf switches. I hear you ask, Why?

The reason for this is that on the newer 1455 VIC card, ports are bundled together by default. Ports 1 and 2 are a bundle and ports 3 and 4 are another bundle. Before we were aware of the updates and keeping our deployment approach consistent with the older APIC appliances, we initially deployed the APIC-L3 appliances using ports 1 and 2 connected to different leaf switches.

However, when using ports 1 and 2 and the default UCS server VIC card settings, we encountered fabric discovery stabilisation issues. We were seeing leaf switches constantly being lost and rediscovered. Through the APIC LLDP interface tool (e.g. #acidiag run lldptool in eth2-1), we noticed that the attached node was flapping between the two upstream leaf switches.

After some investigations, the resolution was to move the connection from port 2 to port 3, which resolved the issue.

“Have you been told to disable the CIMC Port-Channel feature”

(Updated 18/10/2019) We have been made aware that some customers may have been provided with a solution, which is actually not a Cisco supported solution when used with the APIC hardware. If you have in the past been asked to disable the LACP bundling on the APIC-M3/L3 hardware you should look to reach out to Cisco TAC and revert this change at a time that is convenient within your business. This is to ensure that you operating inline with Cisco's supported configuration.

Here is a screen shot of the CIMC VIC Settings that you should look to have re-enabled at the earliest opportunity.