10 Gigabit Ethernet
10 Gigabit Ethernet (10GE, 10GbE, or 10 GigE) is a group of computer networking technologies for transmitting Ethernet frames at a rate of 10 gigabits per second (10×109 or 10 billion bits per second). It was first defined by the IEEE 802.3ae-2002 standard. Unlike previous Ethernet standards, 10 Gigabit Ethernet defines only full duplex point-to-point links which are generally connected by network switches; shared-medium CSMA/CD operation has not been carried over from the previous generations Ethernet standards. Half duplex operation and repeater hubs do not exist in 10GbE.
Like previous versions of Ethernet, 10GbE can use either copper or fiber cabling. However, because of its bandwidth requirements, higher-grade copper cables are required: category 6a or Class F/Category 7 cables for lengths up to 100 meters. The 10 Gigabit Ethernet standard encompasses a number of different physical layer (PHY) standards. A networking device, such as a switch or a network interface controller may have different PHY types through pluggable PHY modules, such as those based on SFP+. At the time that the 10 Gigabit Ethernet standard was developed, interest in 10GbE as a wide area network (WAN) transport led to the introduction of a WAN PHY for 10GbE. The WAN PHY encapsulates Ethernet packets in SONET OC-192c frames and operates at a slightly slower data-rate (9.95328 Gbit/s) than the local area network (LAN) PHY.
The adoption of 10 Gigabit Ethernet has been more gradual than previous revisions of Ethernet: in 2007, one million 10GbE ports were shipped, in 2009 two million ports were shipped, and in 2010 over three million ports were shipped, with an estimated nine million ports in 2011. As of 2012, the price per port of 10 Gigabit Ethernet relative to its one gigabit counterpart still hindered more widespread adoption, even though the price per gigabit of bandwidth enabled by 10 Gigabit Ethernet was already about one-third compared to the bandwidth cost of its one gigabit per second predecessor.
|802.3ae||2002||10 Gbit/s Ethernet over fiber for LAN (10GBASE-SR, 10GBASE-LR, 10GBASE-ER, 10GBASE-LX4) and WAN (10GBASE-SW, 10GBASE-LW, 10GBASE-EW)|
|802.3ak||2004||10GBASE-CX4 10 Gbit/s Ethernet over twin-axial cable|
|802.3-2005||2005||A revision of base standard incorporating 802.3ae, 802.3ak and errata|
|802.3an||2006||10GBASE-T 10 Gbit/s Ethernet over copper twisted pair cable|
|802.3ap||2007||Backplane Ethernet, 1 and 10 Gbit/s over printed circuit boards (10GBASE-KR and 10GBASE-KX4)|
|802.3aq||2006||10GBASE-LRM 10 Gbit/s Ethernet over multimode fiber with enhanced equalization|
|802.3-2008||2008||A revision of base standard incorporating the 802.3an/ap/aq/as amendments, two corrigenda and errata. Link aggregation moved to 802.1AX.|
|802.3av||2009||10GBASE-PR 10 Gbit/s Ethernet PHY for EPON|
|802.3-2015||2015||The latest version of the base standard|
|802.3bz||2016||2.5 Gigabit and 5 Gigabit Ethernet over Cat-5/Cat-6 twisted pair – 2.5GBASE-T and 5GBASE-T|
Physical layer modules
To implement different 10GbE physical layer standards, many interfaces consist of a standard socket into which different PHY modules may be plugged. Physical layer modules are not specified in an official standards body but by multi-source agreements (MSAs) that can be negotiated more quickly. Relevant MSAs for 10GbE include XENPAK (and related X2 and XPAK), XFP and SFP+. When choosing a PHY module, a designer considers cost, reach, media type, power consumption, and size (form factor). A single point-to-point link can have different MSA pluggable formats on either end (e.g. XPAK and SFP+) as long as the 10GbE optical or copper port type (e.g. 10GBASE-SR) inside the pluggable is identical.
XENPAK was the first MSA for 10GE and had the largest form factor. X2 and XPAK were later competing standards with smaller form factors. X2 and XPAK have not been as successful in the market as XENPAK. XFP came after X2 and XPAK and it is also smaller.
The newest module standard is the enhanced small form-factor pluggable transceiver, generally called SFP+. Based on the small form-factor pluggable transceiver (SFP) and developed by the ANSI T11 fibre channel group, it is smaller still and lower power than XFP. SFP+ has become the most popular socket on 10GE systems. SFP+ modules do only optical to electrical conversion, no clock and data recovery, putting a higher burden on the host's channel equalization. SFP+ modules share a common physical form factor with legacy SFP modules, allowing higher port density than XFP and the re-use of existing designs for 24 or 48 ports in a 19" rack width blade.
Optical modules are connected to a host by either a XAUI, XFI or SFI interface. XENPAK, X2, and XPAK modules use XAUI to connect to their hosts. XAUI (XGXS) uses a four-lane data channel and is specified in IEEE 802.3 Clause 48. XFP modules use a XFI interface and SFP+ modules use an SFI interface. XFI and SFI use a single lane data channel and the 64b/66b encoding specified in IEEE 802.3 Clause 49.
SFP+ modules can further be grouped into two types of host interfaces: linear or limiting. Limiting modules are preferred except when using old fiber infrastructure which requires the use of the linear interface provided by 10GBASE-LRM modules.
|Interconnect||AKA||Defined||Connector||Medium||Media type||Wavelength||Max range||Notes|
|10GBASE-USR||Ultra short reach||Appeared in 2011||X2, SFP+||Fiber||Serial multi-mode||850 nm||100 or 150 m||vendor-specific, identical to 10GBASE-SR except for lower performance/reach|
|10GBASE-SR||Short reach||2002||XENPAK, X2, XFP, SFP+||Fiber||Serial multi-mode||850 nm|| OM3: 300 m|
OM4: 400 m
|10GBASE-LR||Long reach||2002||XENPAK, X2, XFP, SFP+||Fiber||Serial single-mode||1310 nm||10 km|
|10GBASE-ER||Extended reach||2002||XENPAK, X2, XFP, SFP+||Fiber||Serial single-mode||1550 nm||40 km|
|10GBASE-ZR||-||XENPAK, X2, XFP, SFP+||Fiber||Serial single-mode||1550 nm||80 km||Not covered by IEEE 802.3ae|
|10GBASE-LX4||2002||XENPAK, X2, SFP+||Fiber||CWDM multi-mode or single-mode||1275, 1300, 1325, 1350 nm|| 300 m (multi-mode)|
10 km (single-mode)
|Costly and complex, mostly replaced by 10GBASE-LRM|
|10GBASE-LRM||Long reach multi-mode||2006||XENPAK, X2, SFP+||Fiber||Serial multi-mode||1310 nm||220 m|
|10GBASE-CX4||2004||XENPAK, X2||Copper||InfiniBand 4X twinaxial 8-pair||-||15 m||Four lanes, each at 3.125 Gbit/s. Larger form factor, bulkier cables and more expensive than SFP+ Direct Attach|
|SFP+ Direct Attach||DA, "10GBASE-CR"||2006||SFP+||Copper||Twinaxial 2-pair||-||15 m||Cheap, low latency, low power|
|10GBASE-T||2006||8P8C||Copper||Class E channel using category 6, Class Ea channel using 6a or 7 twisted pair||-|| 55 m (Class E cat 6)|
100 m (Class Ea cat 6a or 7)
|Can reuse existing cables, high port density, relatively high power|
|10GBASE-KX4||802.3ap||2007||Copper||PCB backplane||-||1 m|
|10GBASE-KR||802.3ap||2007||Copper||PCB backplane||-||1 m|
|10GBASE-PR||802.3av||2009||Fiber||Passive Optical Network||1270 nm/1577 nm||20 km||10G EPON|
There are two classifications for optical fiber: single-mode (SMF) and multimode (MMF). In SMF light follows a single path through the fiber while in MMF it takes multiple paths resulting in differential mode delay (DMD). SMF is used for long distance communication and MMF is used for distances of less than 300 m. SMF has a narrower core (8.3 µm) which requires a more precise termination and connection method. MMF has a wider core (50 or 62.5 µm). The advantage of MMF is that it can be driven by a low cost Vertical-cavity surface-emitting laser (VCSEL) for short distances, and multimode connectors are cheaper and easier to terminate reliably in the field. The advantage of SMF is that it can work over longer distances.
In the 802.3 standard reference is made to FDDI-grade MMF fiber. This has a 62.5 µm core and a minimum modal bandwidth of 160 MHz·km at 850 nm. It was originally installed in the early 1990s for FDDI and 100BaseFX networks. The 802.3 standard also references ISO/IEC 11801 which specifies optical MMF fiber types OM1, OM2, OM3 and OM4. OM1 has a 62.5 µm core while the others have a 50 µm core. At 850 nm the minimum modal bandwidth of OM1 is 200 MHz·km, of OM2 500 MHz·km, of OM3 2000 MHz·km and of OM4 4700 MHz·km. FDDI-grade cable is now obsolete and new structured cabling installations use either OM3 or OM4 cabling. OM3 cable can carry 10GbE 300 meters using low cost 10GBASE-SR optics (OM4 can manage 400 meters) .
To distinguish SMF from MMF cables, SMF cables are usually yellow, while MMF cables are orange (OM1 & OM2) or aqua (OM3 & OM4). However, in fiber optics there is no uniform color for any specific optical speed or technology with the exception being angular physical connector (APC), it being an agreed color of green.
There are also active optical cables (AOC). These have the optical electronics already connected eliminating the connectors between the cable and the optical module. They plug into standard optical module sockets. They are lower cost than other optical solutions because the manufacturer can match the electronics to the required length and type of cable.
10GBASE-SR ("short range") is a port type for multi-mode fiber and uses 850 nm lasers. Its Physical Coding Sublayer 64b/66b PCS is defined in IEEE 802.3 Clause 49 and its Physical Medium Dependent PMD in Clause 52. It delivers serialized data at a line rate of 10.3125 Gbit/s.
| Fibre type|
MMF has the advantage over SMF of having lower cost connectors because of its wider core.
The 10GBASE-SR transmitter is implemented with a VCSEL which is low cost and low power. OM3 and OM4 optical cabling is sometimes described as laser optimized because they have been designed to work with VCSELs. 10GBASE-SR delivers the lowest cost, lowest power and smallest form factor optical modules.
There is a non-standard lower cost, lower power variant sometimes referred to as 10GBASE-SRL (10GBASE-SR lite). This is inter-operable with 10GBASE-SR but only has a reach of 100 meters.
10GBASE-LR ("long reach") is a port type for single-mode fiber and uses 1310 nm lasers. Its Physical Coding Sublayer 64b/66b PCS is defined in IEEE 802.3 Clause 49 and its Physical Medium Dependent PMD in Clause 52. It delivers serialized data at a line rate of 10.3125 GBd.
The 10GBASE-LR transmitter is implemented with a Fabry–Pérot or Distributed feedback laser (DFB). DFB lasers are more expensive than VCSELs but their high power and longer wavelength allow efficient coupling into the small core of single-mode fiber over greater distances.
10GBASE-LRM, (Long Reach Multimode) originally specified in IEEE 802.3aq is a port type for multimode fiber and uses 1310 nm lasers. Its Physical Coding Sublayer 64b/66b PCS is defined in IEEE 802.3 Clause 49 and its Physical Medium Dependent PMD in Clause 68. It delivers serialized data at a line rate of 10.3125 Gbit/s.
10GBASE-LRM allows distances up to 220 metres (720 ft) on FDDI-grade multimode fiber and the same 220m maximum reach on OM1, OM2 and OM3 fiber types. 10GBASE-LRM reach is not quite as far as the older 10GBASE-LX4 standard.
To ensure that specifications are met over FDDI-grade, OM1 and OM2 fibers, the transmitter should be coupled through a mode conditioning patch cord. No mode conditioning patch cord is required for applications over OM3 or OM4.
Some 10GBASE-LRM transceivers also allow distances up to 300 metres (980 ft) on standard single-mode fiber (SMF, G.652), however this is not part of the IEEE or MSA specification.
10GBASE-ER ("extended reach") is a port type for single-mode fiber and uses 1550 nm lasers. Its Physical Coding Sublayer 64b/66b PCS is defined in IEEE 802.3 Clause 49 and its Physical Medium Dependent PMD in Clause 52. It delivers serialized data at a line rate of 10.3125 Gbit/s.
The 10GBASE-ER transmitter is implemented with an externally modulated laser (EML).
Several manufacturers have introduced 80 km (50 mi) range ER pluggable interfaces under the name 10GBASE-ZR. This 80 km PHY is not specified within the IEEE 802.3ae standard and manufacturers have created their own specifications based upon the 80 km PHY described in the OC-192/STM-64 SDH/SONET specifications.
The 802.3 standard will not be amended to cover the ZR PHY.
10GBASE-LX4 is a port type for multimode fiber and single-mode fiber. It uses four separate laser sources operating at 3.125 Gbit/s and coarse WDM with four unique wavelengths around 1310 nm. Its Physical Coding Sublayer 8B10B PCS is defined in IEEE 802.3 Clause 48 and its Physical Medium Dependent PMD in Clause 53.
It allows a range of 300 metres (980 ft) over FDDI-grade, OM1, OM2 and OM3 multimode cabling (all these fiber types are specified to have a minimum modal bandwidth of 500 MHz*km at 1300 nm).
10GBASE-LX4 also allows a range of 10 kilometres (6.2 mi) over SMF.
10GBASE-PR ("PON") originally specified in IEEE 802.3av is a 10G Ethernet PHY for passive optical networks and uses 1577 nm lasers in the down stream direction and 1270 nm lasers in the upstream direction. Its Physical Medium Dependent PMD is specified in Clause 75. Downstream delivers serialized data at a line rate of 10.3125 Gbit/s in a point to multi-point configuration.
10GBASE-PR has three power budgets specified as 10GBASE-PR10, 10GBASE-PR20 and 10GBASE-PR30.
Bi-Directional Single Strand
Multiple vendors have introduced single strand, bi-directional 10 Gbit/s optics capable of a connection functionally equivalent to 10GBase-ER, but using a single strand of fiber optic cable. This is accomplished using a passive prism inside each optical transceiver and a matched pair of transceivers, using a pair of wavelengths such as 1310 nm / 1490 nm or 1490 nm / 1550 nm. Modules are available in varying transmit powers and reach distances ranging from 10 to 80 km.
10G Ethernet can also run over twin-axial cabling, twisted pair cabling, and backplanes.
10GBASE-CX4 was the first 10G copper standard published by 802.3 (as 802.3ak-2004). It uses the XAUI 4-lane PCS (Clause 48) and copper cabling similar to that used by InfiniBand technology. It is specified to work up to a distance of 15 m (49 ft). Each lane carries 3.125 GBd of signaling bandwidth.
10GBASE-CX4 offers the advantages of low power, low cost and low latency, but has a bigger form factor and more bulky cables than the newer single lane SFP+ standard and a much shorter reach than fiber or 10GBASE-T. This cable is fairly rigid and considerably more costly than Category 5 or 6 UTP.
Shipments of 10GBASE-CX4 today are very low. although some network vendors offer CX-4 interfaces which can be used for either 10GBase ethernet or for stacking of switches at (slightly) lower latency. Some examples of combi stacking/ethernet are Dell PowerConnect PCT6200, PCT7000 and the 1G Powerconnect blade switches PCM6220 and PCM6348.
SFP+ Direct Attach
Also known as Direct Attach (DA), Direct Attach Copper (DAC), 10GSFP+Cu, 10GBASE-CR, 10GBASE-CX1, SFP+, or 10GbE Cu SFP cables. Direct Attach uses a passive twin-ax cable assembly and connects directly into an SFP+ housing. SFP+ Direct Attach has a fixed-length cable, typically 1 to 7 m (passive cables) or up to 15 m (active cables) in length, and, like 10GBASE-CX4, is low-power, low-cost and low-latency with the added advantages of using less bulky cables and of having the small form factor of SFP+. SFP+ Direct Attach today is tremendously popular, with more ports installed than 10GBASE-SR.
Backplane Ethernet, also known by its task force name 802.3ap, is used in backplane applications such as blade servers and modular routers/switches with upgradable line cards. 802.3ap implementations are required to operate in an environment comprising up to 1 metre (39 in) of copper printed circuit board with two connectors. The standard defines two port types for 10 Gbit/s (10GBASE-KX4 and 10GBASE-KR) and a 1 Gbit/s port type (1000BASE-KX). It also defines an optional layer for FEC, a backplane autonegotiation protocol and link training for 10GBASE-KR where the receiver can set a three tap transmit equalizer. The autonegotiation protocol selects between 1000BASE-KX, 10GBASE-KX4, 10GBASE-KR or 40GBASE-KR4 operation. 40GBASE-KR4 is defined in 802.3ba.
This operates over four backplane lanes and uses the same physical layer coding (defined in IEEE 802.3 Clause 48) as 10GBASE-CX4.
This operates over a single backplane lane and uses the same physical layer coding (defined in IEEE 802.3 Clause 49) as 10GBASE-LR/ER/SR.
10GBASE-T, or IEEE 802.3an-2006, is a standard released in 2006 to provide 10 Gbit/s connections over unshielded or shielded twisted pair cables, over distances up to 100 metres (330 ft). Category 6a is required to reach the full distance of 100 metres (330 ft) and category 6 may reach a distance of 55 metres (180 ft) depending on the quality of installation, determined only after re-testing to 500 MHz. 10GBASE-T cable infrastructure can also be used for 1000BASE-T allowing a gradual upgrade from 1000BASE-T using autonegotiation to select which speed to use. Due to additional encoding overhead, 10GBASE-T has a slightly higher latency in comparison to most other 10GBASE variants, in the range 2 to 4 microseconds compared to 1 to 12 microseconds on 1000BASE-T. As of 2010 10GBASE-T silicon is available from several manufacturers with claimed power dissipation of 3–4 W at structure widths of 40 nm, and with 28 nm in development, power will continue to decline.
10GBASE-T uses the IEC 60603-7 8P8C modular connectors already widely used with Ethernet. Transmission characteristics are now specified to 500 MHz. To reach this frequency Category 6A or better balanced twisted pair cables specified in ISO/IEC 11801 amendment 2 or ANSI/TIA-568-C.2 are needed to carry 10GBASE-T up to distances of 100 m. Category 6 cables can carry 10GBASE-T for shorter distances when qualified according to the guidelines in ISO TR 24750 or TIA-155-A.
The 802.3an standard specifies the wire-level modulation for 10GBASE-T to use Tomlinson-Harashima precoding (THP) and pulse-amplitude modulation with 16 discrete levels (PAM-16), encoded in a two-dimensional checkerboard pattern known as DSQ128 sent on the line at 800 Msymbols/sec. Prior to precoding, forward error correction (FEC) coding is performed using a [2048,1723]2 low-density parity-check code on 1723 bits, with the parity check matrix construction based on a generalized Reed–Solomon [32,2,31] code over GF(26). Another 1536 bits are uncoded. Within each 1723+1536 block, there are 1+50+8+1 signaling and error detection bits and 3200 data bits (and occupying 320 ns on the line). By contrast PAM-5 is the modulation technique used in 1000BASE-T Gigabit Ethernet.
The modulation used by 10GBASE-T is the basis for the newer and slower NBASE-T standard, implementing a 2.5 or 5.0 Gbit/s connection over existing category 5e or 6 cabling. Cables which will not function at 10GBASE-T speeds may successfully operate at NBASE-T speeds if both devices support.
WAN PHY (10GBASE-W)
The WAN PHY uses the same 10GBASE-S, 10GBASE-L and 10GBASE-E optical PMDs as the LAN PHYs and is designated as 10GBASE-SW, 10GBASE-LW or 10GBASE-EW. Its Physical Coding Sublayer 64b/66b PCS is defined in IEEE 802.3 Clause 49 and its Physical Medium Dependent PMDs in Clauses 52. It also uses a WAN Interface Sublayer (WIS) defined in Clause 50 which adds extra encapsulation to format the frame data to be compatible with SONET STS-192c.
The WAN PHY can drive maximum link distances up to 80 km depending on the fiber standard employed.
- Michael Palmer. Hands-On Networking Fundamentals, 2nd ed. Cengage Learning. p. 180. ISBN 978-1-285-40275-8.
- IEEE 802.3-2012 44.1.1 Scope
- Sharma, Anil (19 January 2011). "LightCounting forecasts CAGR of Over 300 Percent for 10GBASE-T Port Shipments Through 2014". TMCnet. Retrieved 7 May 2011.
- "Dell'Oro press release". Retrieved 29 March 2011.
- "Intel blog about Interop 2011". Retrieved 20 September 2011.
- 10 Gigabit Ethernet still too expensive on servers
- Soz, switch-fondlers: Doesn't look like 2013 is 10Gb Ethernet's year
- "IEEE P802.3ae 10Gb/s Ethernet Task Force". Retrieved 2013-03-19.
- "LightCounting's LightTrends April 2010". Retrieved 3 May 2010.
- "10GbE Optical Component and SFP+ Modules: This Time It's Different by Andrew Schmitt". Retrieved 11 March 2008.
- "The road to SFP+: Examining module and system architectures by Ryan Latchman and Bharat Tailor". Retrieved 15 January 2009.
- 10-Gigabit Ethernet Transceiver Modules Compatibility Matrix
- "The Evolution of Ethernet Nomenclature (Brocade)" (PDF). Retrieved 11 April 2014.
- Practical deployment and management of InfiniBand
- "IEEE 802.3 standard".
- "Optical Fiber and 10 gigabit Ethernet white paper by the 10GEA". Archived from the original on 14 June 2008. Retrieved 1 July 2008.
- "Why choose Multimode fiber? by Corning" (PDF). Retrieved 11 March 2008.
- "10 Gigabit Ethernet over Multimode Fiber by John George" (PDF). Retrieved 10 March 2008.
- "How to tell? MMF or SMF". Retrieved 6 September 2011.
- "Description of Cisco 10G optical modules". Retrieved 3 May 2010.
- "Another Serving of Alphabet Soup — by Intel". Retrieved 4 September 2011.
- "Cisco 10GBASE SFP+ Modules Data Sheet". Cisco Systems. February 2012. Retrieved 2012-05-12.
- "10GBase-LX4 vs 10GBase-LRM: A debate". Archived from the original on 20 July 2009. Retrieved 2009-07-16.
- "Cisco 10GBASE XENPAK Modules". Cisco Systems. November 2011. Retrieved 2012-05-12.
- "Cisco 10GbE optics and 10GBase-ZR".
- "Cisco 10GbE single strand optics" (PDF).
- "Finisar 10GbE single strand optics".
- Dove, Dan."10GBase-CX4 lowers 10G Ethernet cost." Network World. Network World, Inc. 24 May 2004. Web. 19 Dec. 2014.
- "Cables and Transceivers". Arista Networks. Retrieved 21 September 2012.
- "Optcore SFP+ direct-attach cables". Optcore. Retrieved 21 September 2012.
- "HP X242 SFP+ Direct Attach Copper Cable". Hewlett Packard. Retrieved 27 March 2013.
- "IEEE P802.3ap Backplane Ethernet Task Force". Retrieved 30 January 2011.
- "IEEE Standards Status Report for 802.3an". Retrieved 14 August 2007.
- 10GBASE-T for Broad 10 Gigabit Adoption in the Data Center (PDF), Intel, retrieved 2011-12-21
- SWITCHES SWITCH FROM 1000BASE‐T TO 10GBASE‐T NOW (PDF), Teranetics, October 2009, retrieved 2011-12-21
- "Broadcom 10GBASE-T PHY". Retrieved 2 December 2011.
- "PLX Technology, Teranetics 10GBASE-T PHY". Retrieved 11 February 2011.
- "Solar Flare 10GBASE-T PHY". Archived from the original on 7 September 2009. Retrieved 2009-09-05.
- "Aquantia 10GBASE-T PHY" (PDF). Archived (PDF) from the original on 3 December 2008. Retrieved 10 December 2008.
- Hostetler, Jeff. "10GBASE-T – Is 2012 the Year for Wide Adoption?".
- IEEE 802.3-2012 55.1.3 Operation of 10GBASE-T
- Ungerboeck, Gottfried (22 September 2006). "10GBASE-T: 10Gbit/s Ethernet over copper" (PDF). Vienna: Broadcom. Retrieved 7 August 2013.
- "IEEE 802.3 NGEABT Objectives approved by IEEE 802.3, March 12, 2015" (PDF).
- Full text of the IEEE 802.3 standard
- IEEE 802.3 Ethernet Working Group
- Ethernet Alliance website
- University of New Hampshire Interoperability Laboratory 10 Gigabit Ethernet Consortium
- World's First Independent 10GBASE-T Comparative Test Study
- Description of SFP+ Direct Attach server NIC in top-of-rack concept