Monday, October 8, 2012

Cisco Fabric Extender AKA FEX - Nexus 2000 Series

Cisco NEXUS 2000 Series Fabric Extender (FEX)

In today’s post, we are going to dive into new Concept – Fabric Extender. If you are familiar with Traditional Modular Switches AKA Distributed Switching Architecture such as Catalyst 6500/6500 E series or 4500/4500 X series than Fabric Extender is going to be a piece of cake for you. In Distributed Switching Architecture we basically have following components:

Ø  Supervisor Engine – A supervisor engine is basically Mind of switch responsible for management & control plane management.

Ø  Line Card/ Modules - The End Hosts or other devices gets connected on Line Cards/Modules, the basic function of these is to take care of Data Plane/ Forwarding Plane.

Ø  Back Plane/ X-Bar Fabric - The Supervisor Engine/ Engines talk to line cards & Modules using High speed backplane circuitry.

Now in older Distributed Switching platforms all these components reside in a single Chassis. What Cisco did to take this concept a step further is came up with something known as Fabric Extender. A fabric extender basically comes as Cisco Nexus 2000 Series. A fabric extender works as a line card of its parent switch which could be either Nexus 7k or Nexus 5k.

The communication between Parent Switch and Fabric Extender happens through something called Fabric Ports. Let’s see how our new Design with Parent Switch and Fabric Extender Looks like:

So as you can see, the Fabric Extender (FEX) is now the device (Remote Line Cards) to which end hosts gets attached. The FEX further gets connected to Parent Switch which is essentially acting as supervisor engine. It’s the Supervisor Engine where entire configuration is saved which means FEX don’t save any of the configurations locally. Also it’s the parent switch where all forwarding decisions are made. Which means if HOST A on FEX 1 wants to talk to another HOST B connected on same FEX 1, the traffic has to go to parent Switch and sent back towards the FEX as shown in diagram below:

Which is obviously not an idle scenario though but this is the way FEX has been designed. But on the other hand introduction of FEX offers us couple of benefits as well:

   Ø  Reduce Cable Runs
   Ø  Reduces Management Point ( Since Parent Switch Controls all FEXs)
   Ø  Reduces IOS Management & Standardization Load (The FEX always Runs same NX-OS as Parent SW, FEX gets shipped with No NX-OS Image. When we connect FEX to Parent switch and provision it, it downloads the NX-OS from parent switch and uses it)
   Ø  Enables Parent Switch (7K/5K) to become a high density access layer switch
   Ø  STP Free Access Layer ( The fabric links between Parent Switch & FEX runs no STP, on the flip side FEX Host Interfaces(HIF) cannot be used to connect any further switches or devices running Spanning-Tree. BPDU Guard feature is by default enabled on HIF and cannot be disabled. Which means as soon as you plug any device to the FEX running STP, the port will get error disabled)
   Ø  QOS & Security Management from Parent Switch
   Ø  All Troubleshooting From Parent Switch
  Ø Less number of OOB connection

    Since in most of the real world designs you would want to Pair your FEX with Nexus 5K, so we will be discussing basic FEX configuration from Nexus 5K perspective:

   The official NX-OS version in CCIE DC lab for Nexus 5548 is 5.1(3) & FEX models available in lab will be 2232 & 2224.

   Let’s review the topology and relative configuration now:

  Step 1 > Enable FEX feature set on Nexus 5k :
                       5k(config)# feature fex
  Step 2 > Create a Port Channel :
                       5k(config)# interface port-channel 101
  Step 3 > Configure Port Channel to Act as Fabric Link:
                       5k(config-if)# switchport mode fex-fabric
  Step 4 > Assign Associate ID to Remote FEX (From range 101 – 199)
                       5k(config-if)# fex associate 101
  Step 5 > Map Fabric Interfaces to Port Channel:
                       5k(config)# interface e1/1 - 4
                       5k(config-if-range)# channel-group 101
    Once this configuration will be done, we should be ready use FEX. The associate ID gets prepended to FEX interface for identification. For example if FEX port were like : E1/1, E1/2…., After association ID 101 assigned to FEX, on parent switch the ports will now appear as: E101/1/1, E101/1/2. Now any configuration you apply under these ports will be pushed to FEX itself.

   Couple of verification commands:
   Ø  sh fex
   Ø  sh fex detail
   Ø  sh interface status fex  = sh interface status (in regular IOS)

       Couple of limitations with FEX :
   > FEX Doesn't support Private VLANs
   > Any port available on FEX cannot be SPAN destination port
   > FEX Model like 2148 Can't be associated with Nexus 7K
   > So far only FEX model 2232PP supports FCOE on FEX ports 
   > On Nexus 7k, the non-default VDC gets access to FEX feature only if  
          feature is installed and enabled under default VDC
   >  In Nexus 7-K all the uplinks and host ports of a Fabric Extender 
            belong to a single VDC. The ports cannot be allocated or split 
            among multiple VDCs.
   > Nexus 7k F1 modules are though Layer 2 cards, but still doesn't support 
   > You can configure the Fabric Extender host interfaces as edge 
          ports only.The interface is placed in an error disabled state if a   
          downstream switch is detected.
   > The Cisco Nexus 2148 Fabric Extender does not support frames 
            with the dot1p vlan 0 tag

  And here is most interesting one. Just take a look at Pic below.

  Actually you are looking at Back (rear) side of Nexus 2000 FEX, 
   and default Air-flow is from front to back.

Deepak Arora


Anonymous said...

The new version NX-OS you allow you to disable BPDU Guard feature HIF.

Deepak Arora said...

Do you have any reference from the Documentation Standpoint confirming it ?

rvijaybhargav said...

Hi Deepak,
Thanks for the wonderful post on FEX. I was justing checking and you have mentioned that BPDU guard will be enabled by default and any STP related devices connected to that will move to "err-disabled" mode. I was just testing the same but was able to observe a VM connected to FEX and the port was trunk. COnfig are as below

XXXXXXX# sh interface Eth101/1/9 switchport
Name: Ethernet101/1/9
Switchport: Enabled
Switchport Monitor: Not enabled
Operational Mode: trunk
Access Mode VLAN: 156 (XXXXXX)
Trunking Native Mode VLAN: 1 (default)
Trunking VLANs Allowed: 150-154,156,158-159,204,901-905,1101
Pruning VLANs Enabled: 2-1001
Voice VLAN: none
Extended Trust State : not trusted [COS = 0]
Administrative private-vlan primary host-association: none
Administrative private-vlan secondary host-association: none
Administrative private-vlan primary mapping: none
Administrative private-vlan secondary mapping: none
Administrative private-vlan trunk native VLAN: none
Administrative private-vlan trunk encapsulation: dot1q
Administrative private-vlan trunk normal VLANs: none
Administrative private-vlan trunk private VLANs: none
Operational private-vlan: none
Unknown unicast blocked: disabled
Unknown multicast blocked: disabled

XXXXXX# sh cdp neighbors interface Eth101/1/9 detail
Device ID:edcucp01
System Name:
Interface address(es):
IPv4 Address:
Platform: VMware, Capabilities: Host
Interface: Ethernet101/1/9, Port ID (outgoing port): eth0
Holdtime: 130 sec

Linux 2.6.18-194.26.1.el5PAE #1 SMP Fri Oct 29 14:28:58 EDT 2010 CCM:

Advertisement Version: 2
Duplex: full


Deepak Arora said...

Hi Vijay,

Check if your VM switch is running STP or not, since it will send BPDUs out towards FEX only in that case.

Or on Nexus try :

sh spanning-tree int e101/1/9 detail | include BPDU

rvijaybhargav said...

Ya Deepak, you were right. I am not receving any BPDU's. Thanks :)

mahasiswa teladan said...

hi..Im student from Informatics engineering, this article is very informative, thanks for sharing :)

Christophe said...

The documentation says:
"CDP is not supported for the Cisco Nexus 2000 Series Fabric Extender".

How did you manage to get CDP informations on port Eth101/1/9?

Morris Gomes said...

I was looking all over the net trying to find out more on the FEX extenders and new to the 5K switches. This is the best explaination so far!!! thanks

Morris Gomes

Anonymous said...

Thanks Deepak. This article was very helpful in understanding the FEX feature.

deeapkbansal said...



Deepak Arora said...

That IP Range is used to establish out of band ip connectivity between FEX and Parent Switch over which NX-OS from Parent is downloaded over FEX.

Kal said...

Great overview of FEX architecture, much appreciated

Jnkevin Kevin said...

Hello Deepak,

I want to get a confirmation from you.

To connect a parent switch(5672UP) to a FEX (e.g 2348 UPQ), should I use the fabric ports only(yellow ports) or can the other 48x10Gb ports be used ?

Thanks for your clarification.