Saturday, March 26, 2016

Clos Fabrics AKA Spine & Leaf Architecture

Let's start the series with discussion of CLOS Fabrics AKA Spine & Leaf Architectures.




Now CLOS design is not fundamentally new, but most of the Network Engineers were not talking about it till recent times (Well...this is true to an extent). So as Network Engineer should you really care ?




Well you should start by asking why CLOS in first place ?

The major problem that CLOS fabric solves is about solving scalability issues. While scalability is a matter of context, it's not necessary that everyone needs or to be precise going too far about it.

Also CLOS fabric also doesn't define your Layer2 - Layer 3 boundaries itself. So you are pretty much dependent upon what works best for you from vendor implementation perspective while keeping your overall goal in mind. Now in theory Layer 3 Fabrics scale much better than Layer 2 Fabric. Here are some questions/Things you figure out about CLOS if you decide to go for it :

- What is the scale that you got to deal with ?
- What are technical and business requirements ?
- Your DC traffic is mostly east-west or north-south ?
- How you can minimize the state of the Core (Spine) to minimum ?
- How flooding works in your fabric ?
- How multicast is handled in fabric ?
- Where to define Layer2-Layer 3 boundary ?
- Your network is going to multi vendor now/In future ?
- How you gonna manage and monitor such large network ?
- How you gonna introduce security & Services such as Load Balancer ?
- How you gonna connect to external world ? (Border Spine Vs. Border Leaf) 
- Define you convergence requirements 
- You gonna need single or multi stage CLOS ?
- Your over subscription ratio ? (Usually 3:1 is good for most part)
- Understand your failure domains and impact they may have
- Do you need Spine to Spine or Leaf to Leaf connections to mitigate some of     
   failure scenario ?
- If you are going with Layer 3 fabric, is it going to be good idea to use 
   summarization ?
- EBGP vs IBGP (Also RR placement) in Layer 3 fabric ?


Even as an example, Cisco's famous buzzword these days ACI (Application Centric Infrastructure ) also uses Spine & Leaf design. It uses BGP EVPN (Some secret souce but soon EVPN will be there too) control plane and on top of which it uses VXLAN as Data Plane. So between Spine & Leaf (Single Stage) it uses Layer 3 fabric. The entire fabric is managed with a centralized command and control system called Cisco APIC Controller. With ACI you can go as far as 6 Spines at the moment and all services (e.g. load balancer), firewalls, external connectivity gets terminated on Leaf switches. For server redundancy (Bare Metal Or Virtual ) it uses our old friend Virtual Port Channel (vPC) but this time doesn't require directly connected interfaces among leaf switches for peer link and peer keep alive link functions. 

Cisco ACI is kind of build around another buzz word that you hear more often these days called SDN (Software Defined Networks). Now whether it fits into true SDN definition or not needs another discussion :).

In the mean while below is the list of URLs which you may find very handy to get started with CLOS:

http://packetpushers.net/podcast/podcasts/datanauts-011-understanding-leaf-spine-networks/

https://code.facebook.com/posts/360346274145943/introducing-data-center-fabric-the-next-generation-facebook-data-center-network


http://www.networkworld.com/article/2226122/cisco-subnet/clos-networks--what-s-old-is-new-again.html

http://searchdatacenter.techtarget.com/feature/The-case-for-a-leaf-spine-data-center-topology

http://searchdatacenter.techtarget.com/answer/Whats-the-best-data-center-network-topology

http://searchdatacenter.techtarget.com/feature/Data-center-network-design-moves-from-tree-to-leaf

http://www.excitingip.com/4490/distributed-coreleaf-spine-network-architecture-an-intro/

http://ieeexplore.ieee.org/xpl/login.jsp?tp=&arnumber=4448982&url=http%3A%2F%2Fieeexplore.ieee.org%2Fiel5%2F90%2F4359146%2F04448982.pdf%3Farnumber%3D4448982

http://etherealmind.com/which-network-topology/

http://packetpushers.net/network-topologies/

http://blog.ipspace.net/2014/04/security-in-leaf-and-spine-fabrics.html

http://www.thenetworkingdom.net/bgp-clos-networks/

http://thenetworksurgeon.com/cisco-spine-and-leaf-architecture-discussion-nexus-5500-vs-6001/

http://blog.ipspace.net/2012/04/full-mesh-is-worst-possible-fabric.html

http://conferences.sigcomm.org/sigcomm/2015/pdf/papers/p183.pdf

http://conferences.sigcomm.org/co-next/2013/program/p49.pdf


https://www.nanog.org/meetings/nanog55/presentations/.../Lapukhov.pdf


http://www.juniper.net/us/en/local/pdf/whitepapers/2000565-en.pdf

https://cumulusnetworks.com/blog/routed-vmotion-why/




HTH...
Deepak Arora
Evil CCIE

5 comments:

saki said...

thanks for shared wonderful information of giving best information.its more useful and more helpful. great doing keep sharing
CCNA Training in Chennai

Aashisiva said...

Great articles, first of all Thanks for writing such lovely Post! Earlier I thought that posts are the only most important thing on any blog. But here at Shoutmeloud I found how important other elements are for your blog.Keep update more posts..

J2ee Training institute in Chennai

Kalpu Kvb said...

This blog is having the general information. Got a creative work and this is very different one. We have to develop our creativity mind. This blog helps for this. Thank you for this blog. this is very interesting and useful.
PPC Services Chennai

Kalpu Kvb said...

This blog explains the details of most popular technological details. This helps to learn about what are all the different method is there. And the working methods all of that are explained here. Informative blog.
PPC Services Chennai

Anonymous said...

great post however, ACI doesnt use EVPN for mac advertisement. It uses a proprietary protocol for endpoint advertisement. i think that shud be corrected.