Building a Big Data Backbone: Part 2
31 May 2017
News
Over this two-part blog, we reflect on some of the design decisions faced by data centre designers as fibre optic technology evolves to meet the challenges of a Big Data World.



In part one we looked at how demand is driving a shift in Data Centre architecture and how optical transceiver technology developments has become a key enabler for hyper-scale data centres.  In Part 2 we consider various aspects of the fibre link and how they are changing from the fibre cable itself, the optical transceiver modules, transmission speeds, associated standards, distances and how they are becoming relevant to new build and legacy installations for all data centres.

A Fitting Form Factor
Independent of IEEE standards, several MSA’s (Multi-source agreements) have been produced by leading manufacturers. One such MSA which supports single mode at 100G is known as 100G PSM4 MSA (100G Parallel Single Mode 4 lanes) This has become popular due to its wide manufacturer support and was early to market (2014). It has been deployed mainly as 100GBASE-PSM4. Using widely adopted QSFP28 form factor transceivers it is also flexible in that 100G can be split into 4 x 25G breakout channels.  PSM4 has been a proving ground for silicon photonics, an evolving technology which is bringing the cost of single mode optics for the data centre down. Silicon photonics in itself is a big topic but is not expected to be a disruptive technology for several years as it competes with established Indium Phosphide (InP) & VCEL optics.  

Whilst QSFP28 has been the predominant form factor for 100G deployment, last year another MSA brought forward its successor, the QSFP-DD (Quad small form factor pluggable - Double Density) which will help to accelerate the adoption of 200G & 400G. In March this year, the specification was announced. This double density module will employ 8 lanes that operate up to 25 Gbps NRZ modulation or 50Gbps PAM4 modulation providing 200 Gbps or 400Gbps aggregate. The new form factor will in summary support:
3m of passive direct attached copper cables (DAC),
100m over Multimode fibres (MMF)
500m over Singlemode fibres (SMF)
Backwards compatibility with QSFP transceivers from 40G to 200G
For transceivers and active copper or active optical cable assemblies, this includes those defined by Ethernet, Fibre Channel, and InfiniBand.

Looking after Legacy
Whilst it is looking like single mode fibre will dominate hyper-scale &  new cloud data centres, the majority of legacy data centres in the UK are equipped with VCSEL based 850nm short range transceivers. 10G SFP+ is the dominant form factor connected through duplex LC OM3 or OM4 fibre via cassettes and combined onto 12 fibre MTP trunks. These provide mostly 10G connections and also a lot of copper 1Gig connections in the HDA (Horizontal Distribution Area). This existing infrastructure can migrate using parallel optics to 40Gig or 100G over short distances (100m OM3 / 150m OM4) which are adequate for most data centres.  Transceiver modules aggregate multiple 10G signal paths onto one 40G or 100Gconnection. This has increased the popularity of QSFP+, CXP and other form factors. 

However the number of fibre MTP trunks, especially if re-configuring from a 3-tier to 2-tier architecture is expensive as each 100G link requires one MTP fibre trunk, using either 20 fibres (100GBASE-SR10) or 8 fibres (100GBASE-SR4). In the case of 100GBASE-SR4 using 25G per fibre, only 100m reach is possible for OM4. Breaking out from 100G also requires MTP to LC breakout cables which also adds expense. The option of migrating further to 400G for which some data centres are already provisioning is hence practically unrealistic using single wavelength VCEL lasers.

Last June we saw the publication of ANSI/TIA 492-AAAE standard for wideband multimode fibre (WBMMF) or OM5.  OM5 standardised fibre is 50microns just like OM4 and is backwards compatible but designed to carry multiple short wavelength signals (850 to 953 nm) that can be aggregated for high bandwidth applications i.e. short wave division multiplexing (SWDM). This could enable a data centre to retain its installed OM3/OM4 duplex fibre infrastructure or supplement with OM5 where longer reach is required. In 2015 the SWDM alliance was created to promote this technology. With SWDM4 (SWDM+ four pass-bands i.e. four signal paths on a single multimode fibre), an optical transceiver with a standard QSFP28 interface can drive 100Gbps over a legacy single pair of multimode fibres.

However, OM3 and OM4 fibre will incur a loss penalty * at the upper pass band, restricting reach. OM5 provides greater reach, typically 150m compared with 70/100m for standard OM4/OM5. (* Not all OM3/OM4 fibres are born equal. It is worth checking the specification e.g. Effective Modal Bandwidth EMB with the supplier if contemplating running SWDM optics with it) A possible drawback of SWDM over duplex MM fibre is that it cannot readily be broken out e.g. to 10G or 25G in the same way as parallel optics.  SWDM4 VCSEL optics are expected to be available this year but there are already existing proprietary WBMMF solutions on the market. Cisco’s 40G BiDi transceiver solution (40GBASE-SR-BiDi) allows re-use of duplex multimode fibre for a 40G connection over OM4. The BiDi transceiver employs two wavelengths (850nm & 900nm) transmitting in opposite directions over each fibre.  

Looking ahead it is anticipated that with tighter, faster VCSEL optics, and with PAM modulation it will be possible to achieve 200G over multimode.  Some are sceptical whether the cost of 100G SWDM4 optical transceivers and associated OM5 fibre cable will prove to be a compelling business decision. This remains to be seen but avoiding the business disruption in switching out existing fibre infrastructure must make SWDM worth looking at for legacy data centres. One thing for sure is as more data migrates onto a single circuit, each circuit becomes ever more mission-critical.

References
http://www.ieee802.org/3/cd/public/Sept16/pimpinella_3cd_01a_0916.pdf
http://blog.leviton.com/om5-fiber-good-solution-data-center
http://blog.siemon.com/infrastructure/om5-recognized-by-tia-and-iec-what-now
http://www.belden.com/blog/datacenters/will-silicon-photonics-transform-data-centers.cfm http://www.belden.com/blog/datacenters/a-cost-effective-solution-short-wavelength-division-multiplexing-swdm.cfm
http://www.swdm.org/swdm-brownfield-use-case/ http://www.mellanox.com/blog/2017/03/100g-psm4-the-most-configurable-lowest-cost-single-mode-transceiver-available/
http://www.cisco.com/c/en/us/solutions/service-provider/global-cloud-index-gci/white-paper-listing.html
http://ix.br/ixforum/9/slides/ixbr9-finisar.pdf
http://www.qsfp-dd.com/
http://psm4.org/