An Engineer by Heart !!!
A Dreamer, A Pioneer, A Blogger.
A Network Engineer Trying to overtake the world with his network engineering skills :)
Opinions expressed here are solely my own and do not express the views or opinions of my Present or Past employer.
As the Cisco Live 2013 at Orlando is keep going on, Cisco just announced a New Baby into Catalyst family as Cat 6800. Based on Cat 6500 DNA with better performance (Perhaps matching Nexus) and backward compatibility with Cat 6500 series line cards the new Baby is meant to replace current Cat 6500 and give an option to people which probably don't want to move on to Nexus yet.
1. Go to the MDS physical interfaces, shut them down 2. Need to ensure the individual ports are dedicated - 'switchport rate mode dedicated' otherwise will error out on port-channel configuration. Can also configure 'switchport mode e' 3. Add ports to port-channel - 'channel-group X' 4. Go to the port-channel, apply the configuration ('switchport mode e' and 'switchport rate mode dedicated'. 5. If required, restrict the port-channel with the only the necessary VSANs 6. Add the port-channel to the VSAN database (optional - again depends on requirements) 7. No shut the port-channel and then physical ports
* Make use of 'show interface fc x/y' and verify config - look at: Admin port mode, trunk mode, port mode, port vsan, VSANs allowed (if trunking). * Ensure that these settings match on both sides (either MDS to N5K or UCS)
For F/TF port-channel, steps that are slightly different from above:
1. Enable MDS features (such as npiv / fport-channel-trunk - if making a TF port-channel) 2. For F ports, can leave in either shared or dedicated mode. Additionally, configure the physical interfaces as 'switchport mode f', otherwise will error out as generally they default to FX ports 3. Instead of 'switchport mode e' on the port-channel, obviously make it an F port
If link fails to come up and using NPV-NPiV, useful to make use of 'show npv status', will also identify if you've forgotten to enable NPiV upstream, etc.
# Note: One copy of FC services runs on per VSAN basis which means One domain id can be used for multiple VSANs but it doesn't affect the design. It's more like using one OSPF router id for multiple ospf processes. But each process populates a separate Datbase which is completely unrelated from each other.
* Changing Port Type to FC in 5k ================================
slot 1 or 2 port 25 - 32 type fc
+ Requires Reload
# " feature fcoe " needs to be turned on separately in order to enable basic FC protocol which also further enables actual FCOE. Though NPV for FCOE requires separate license.
+ If port type has been changed to FC but feature fcoe is not enabled, the FC ports will not show up in "sh int brief" output. Also we will not be able to go into FC interface configuration mode.
+ The SFP for storage connecivity is different from SFP for Ethernet connection from 5K standpoint
* FC Auto Negotiation =====================
> FC Auto Negotiation is enabled by default
> Auto Negotiation Determines:
+ Port Type [ switchport mode ]
+ Trunk Mode [ switchport trunk mode ]
+ Speed [ switchport speed <1000> ]
- From real world standpoint, Port Types etc should be configured manually
* MDS Auto-Negotiation ======================
- MDS port types are by default FX which means it can figure out if it needs to be F or FL 1000> port mode during auto-negotiation
- MDS usually can't negotiate E ports so ISL must always be configured manually.
- In MDS E port requires dedicated rate-mode [switchport rate-mode dedicated ]
+ In FC network it's not only SFP type that will control the link speed but also Optical cable should be of same speed in order to match it
+ In FC network there is something known as B2B or Buffer To Buffer credit which is analogous to Flow Control is Ethernet Networks.B2B works per link basis and on end to end basis between initiator & target
* FC Trunk Protocol ===================
- Trunk protocol (TP) in FC is analogous to DTP in ethernet and is enabled by default
- Verification - sh trunk protocol
- To disable TP - switchport trunk mode on (Default - auto)
But keep in mind there is no command called "switchport mode te" to statically configure TE port
After couple of months break now I am back on my CCIE DC preparation track. Since I was pretty much done with Nexus at Initial level and also did couple of Nexus Installations at work along with OTV deployment I thought to lets start with storage now. Since I had no experience with storage at any level earlier so I thought to start with baby steps as of now and catch up the pace later.
When it comes to storage there is no better choice other to start learning one of Storage Core Protocol known as Fibre Channel AKA FC.
So I did some initial FC study yesterday and compiled some notes for reference. Hope you guys will find it helpful to begin with storage.
> FC is a Protocol Stack primarily used to send SCSI commands over the SAN. But we can also run IP/ARP over FC
# FC Suppported topologies
1. P2P (FC-P2P) 2. Arbitrated Loop (FC-AL) similar to Token Ring 3. Switched Fabric (FC-SW) similar to Ethernet
# FC Port Types
1. Node Port AKA N Port - End host (Target or initiator) in P2P or Switched Fabric 2. Node Loop Port or NL Port - Similar to N port but in Arbitrated Loop Topology e.g. > JBOD Connectivity 3. Fabric Port or F Port - Switch port connecting to Node Port 4. Fabric Loop or FL Port - Switch port connecting to Node Loop Port 5. Expansion Port or E Port aka ISL = Link between two switches 6. Trunk Expension Port or TE Port = Link between two switches to carry VSANs similar to DOT1Q trunk in LAN
# Fibre Channel Addressing
1. Physical Address = World Wide Names (WWNs) is equal to MAC in Ethernet = 8 Bytes 2. Logical Address = FC Identifier (FC ID) is equal to IP address = 3 Bytes
* WWN is subdivided into two parts
1. World Wide Node Name (WWNN)
- Switch, Server or Disk's Physical Address
2. World Wide Port Name (WWPN)
- Switch, Server or Disk's Port's Physical Address
E.g. > A HBA with multiple interfaces. HBA will have one WWNN and Each HBA port will have it's own WWPN
Note - WWN is not used in Data Plane
* FCID is subdivided into three fields
1. Domain ID
- Each switch gets a separate domain id
2. Area ID
- Group of ports on switch have an area id
3. Port ID
- End station connected to switch gets a Port ID
> Domain ID is automatically assigned by Principal Switch but can also be manually assigned. Principal switch is analogous to STP root bridge in Ethernet world.
# Fabric Shortest Path First (FSPF) is protocol in SAN used to route traffic between Switches based on FCID's Domain ID
FSPF support ECMP and used Dijkstra Algorithm like OSPF & ISIS and by default enabled on SAN Switch as Service
# FC supports SAN Port Channel using Port Channel Protocol (PCP) roghly analogous to 802.3ad
# In Fibre Channel network all end hosts must first register with the control plane of fabric before sending any traffic
# Fabric Registration has three parts
1. Fabric Login (FLOGI) 2. Port Login (PLOGI) 3. Process Login (PLRI)
> During FLOGI, the N Port tells switch's F port that it wants to register
> Once Switch gets Flogi Request, it learns the WWNN and WWPN of node
> Later switch assigns FCID to node
> End to End Login between N Ports
> Initiator (End Device such as server) tells target (Storage Array) that it wants to talk
> Used for applications such as End To End flow control
> Upper Layer protocol login negotiation between N ports
# Fiber Channel Name Server (FCNS) is analogous to ARP cache and helps in FLOGI.
> FCNS is used to resolve WWN to FCID
sh fcns database
sh flogi database
> Zoning controls which initiators talks to which targets and alalogous to ACL in IP World
- Associates WWNs, FCIDs, aliases etc to control who can talk to whom
Virtual SANs (VSANs) ====================
> VSANs are roughly analogous to VLANs in Ethernet world
> VSANs are used to create small SAN separations within large SAN. Also can be used to isolate the management and failure domain of the network
> Creating a VSAN creates a separate/new set of Fabric Services like FLOGI, FCNS, ZONING, Aliases etc