Data Center Cabling Challenges Past, Present & FutureJONATHAN JEW – JEW@J-AND-M.COM

My Past ….

35 years 100+ data center projectsSmall computer rooms to mega data centers

Data Center and Telecom Cabling Standards

17 years TIA – US & Canadian standardsBICSI – International Best Practices ISO/IEC – International standards

In the beginning … 1970’s-1980’s Computer rooms commonly

located in converted office space Almost entirely mainframes and

minicomputers Communications cabling

No LAN or SAN Proprietary channel cabling Proprietary communications

cabling Standards-based CCITT X.21, CCITT

V.35, EIA RS-232 cabling 9600 bps common speed 56000 bps was a fast speed

Large Proprietary Point-to-Point Cabling

IEEE 802 LAN Standards

IEEE 802.3 Ethernet (mostly DEC) and IEEE 802.5 Token Ring (mostly IBM)

Higher Speeds – Initially 10 MbpsEthernet and 4 Mbps Token Ring

Smaller cable diameters Standards-based cabling Addressed large proprietary and

relatively slow cabling in data centers

TIA-568 Cabling Standards TIA/EIA-568-B:1995 defined

Category 3, 4, 5 cables Cat 3 (up to 10 Mbps) Cat 4 (up to 16 Mbps) Cat 5 (up to 100 Mbps) Replaced coaxial cable used for

Ethernet and Type 1 cable used for Token Ring

Solved single-use cabling problem - generic cabling usable in data centers for wide range of other applications (WAN, console, terminals, voice) and further reduced cable sizes

TIA-606 Administration Standard ANSI/TIA-606:1993 Provided standard for labeling and administration of structured cabling systems

Easy to troubleshoot and make changesDifficult to troubleshoot and make changes

All ports and cords labeled(with near and far end port IDs)

Only one label (ne pas disconnect )

Impact of Documentation, Labeling, & Equipment Refresh

Labeling and documentation substantially reduced outage time (addressed in TIA-606)

Equipment life cycle refresh reduced frequency of failures

TIA-942 Telecom Infrastructure Standard for Data Centers ANSI/TIA-942:2005 First structured cabling standard for data centers Addressed a wide range of other subjects

Ceiling & door heights Lighting Floor loading Cabinet design and placement Temperature & humidity Separation of power and twisted pair Circuit distances Site selection Tier/Rating

Corresponding ISO/IEC 11801-5 only addresses cabling

Point-to-point unstructured cablingH

eate

r - A

/C

Heater - A/CHeater - A/C Heater - A/C

247-7538

HALONABORTHALON

RELEASE

247-7543

HALONPANEL

LOCKER

TAPE RACK

NETWORKCABINET

H5

NETWORKCABINET

H4

NETWORKCABINET

H3

NETWORKCABINET

H2

NETWORKCABINET

H6

NETWORKCABINET

H7

NETWORKCABINET

H1

NETWORKCABINET

G4

NETWORKCABINET

G3

NETWORKCABINET

G2

NETWORKCABINET

G1

NETWORKCABINET

G5

TABL

E

TAPE RACK

NET

WO

RK

CAB

INET

I 8

NET

WO

RK

CAB

INET

I7

NET

WO

RK

CAB

INET

I6

NET

WO

RK

CAB

INET

I5

NET

WO

RK

CAB

INET

I4

NET

WO

RK

CAB

INET

I3

NET

WO

RK

CAB

INET

I2

NET

WO

RK

CAB

INET

I 1

TAPE RACK

NETWORKCABINET

H8

NETWORKCABINET

G6

NETWORKCABINET

G7

NETWORKCABINET

F4

NETWORKCABINET

F3

NETWORKCABINET

F2

NETWORKCABINET

F5

LANCABINET

F1

NETWORKCABINET

F6

NETWORKCABINET

E5

NETWORKCABINET

E4

NETWORKCABINET

E3

NETWORKCABINET

E2

NETWORKCABINET

E1

NETWORKCABINET

E6

SAACABINET

D2

NETWORKCABINET

E7IBMCAB

SAACABINET

D1

NETWORKCABINET

B5

NETWORKCABINET

B4

NETWORKCABINET

B3

NETWORKCABINET

B2

NETWORKCABINET

B1

NETWORKCABINET

B6

NETWORKCABINET

C5

NETWORKCABINET

C4

NETWORKCABINET

C3

NETWORKCABINET

C2

NETWORKCABINET

C1

NETWORKCABINET

C6

NETWORKCABINET

B7

CD ROMSCANNER

AT&TDEFINITY

UNIT

HPPRINTER

NETWORKCABINET

A6

NETWORKCABINET

A5

NETWORKCABINET

A4

NETWORKCABINET

A3

NETWORKCABINET

A2

TABLE247-7589

TABLE

CISC

O

RO

UTER

CO

NSO

LE

F/X

IBMDASD

IBMDASD

IBMAS-400

IBMAS-400

IBMS-390R34

IBM3746-900

IBM3745-17A

IBM3745-L13

IBM

3745-41D

IBM3745-A11

POM

ON

AA/

C U

NIT

#8

STO

RAG

EC

ABIN

ET

DES

K

DESKSTORAGECABINET

PAC

BEL

LC

ABIN

ETA 1

3174CABINET

TABLE STORAGECABINET

STORAGECABINET

LIEBERTPDU-C

3174CABINET

3174CABINET

3174CABINET

3174CABINET

3174CABINET

POMONAA/C UNIT

#6

POMONAA/C UNIT

#7

FILECABINET

FILECABINET

FILECABINET

FILECABINET

G H I J K L M N O P Q R S T U V W X Y Z AA AB AC AD AE AF AG AH AI AJ AK AL AM AN AO AP AQ AR ASC D E FBA

2

1

3

4

5

6

7

8

9

10

11

12

13

14

15

16

NETWORK

INSTALL A CABLE WHEN YOU NEED IT (SINGLE-USE, UNORGANIZED CABLING)

Entrance Room(Carrier Equip &

Demarcation)

Access Providers

HDA(LAN/SAN/KVM

Switches)

EDA(Rack/Cabinet)

ComputerRoom

HDA(LAN/SAN/KVM

Switches)ZDA

Work Areas in Offices, Operations

Center, Support Rooms

TR(Office & Operations

Center LAN Switches)

MDA(Routers, Backbone LAN/SAN Switches, PBX, M13 Muxes)

HDA(LAN/SAN/KVM

Switches)

EDA(Rack/Cabinet)

EDA(Rack/Cabinet)

Backbone cabling

Horizontal cabling Horizontal cabling Horizontal cabling

Horizontal cabling

Backbone cabling

Backbonecabling

Horizontal cabling

Access Providers

HDA(LAN/SAN/KVM

Switches)

EDA(Rack/Cabinet)

Horizontal cabling

TIA-942Data CenterTopology

Structured cabling – faster deployment, easier to manage & troubleshoot

Hea

ter -

A/C

Heater - A/CHeater - A/C Heater - A/C

247-7538

HALONAB ORTHALON

RELE ASE

247-7543

HALONPA NEL

LOCKE R

TAPE RACK

NETWORKCABINET

H5

NETWORKCABINET

H4

NETWORKCABINET

H3

NETWORKCABINET

H2

NETWORKCABINET

H6

NETWORKCABINET

H7

NETWORKCABINET

H1

NETWORKCABINET

G4

NETWORKCABINET

G3

NETWORKCABINET

G2

NETWORKCABINET

G1

NETWORKCABINET

G5

TABL

E

TAPE RACK

NETW

ORK

CABI

NET

I8

NETW

ORK

CABI

NET

I7

NETW

ORK

CABI

NET

I6

NETW

ORK

CABI

NET

I5

NETW

ORK

CABI

NET

I4

NETW

ORK

CABI

NET

I3

NETW

ORK

CABI

NET

I2

NETW

ORK

CABI

NET

I1

TAPE RACK

NETWORKCABINET

H8

NETWORKCABINET

G6

NETWORKCABINET

G7

NETWORKCABINET

F4

NETWORKCABINET

F3

NETWORKCABINET

F2

NETWORKCABINET

F5

LANCABINET

F1

NETWORKCABINET

F6

NETWORKCABINET

E5

NETWORKCABINET

E4

NETWORKCABINET

E3

NETWORKCABINET

E2

NETWORKCABINET

E1

NETWORKCABINET

E6

SA ACABINET

D2

NETWORKCABINET

E7IBMCAB

SA ACABINET

D1

NETWORKCABINET

B5

NETWORKCABINET

B4

NETWORKCABINET

B3

NETWORKCABINET

B2

NETWORKCABINET

B1

NETWORKCABINET

B6

NETWORKCABINET

C5

NETWORKCABINET

C4

NETWORKCABINET

C3

NETWORKCABINET

C2

NETWORKCABINET

C1

NETWORKCABINET

C6

NETWORKCABINET

B7

CD ROMSCANNE R

AT&TDEF INITY

UNIT

HPPRINTER

NETWORKCABINET

A6

NETWORKCABINET

A5

NETWORKCABINET

A4

NETWORKCABINET

A3

NETWORKCABINET

A2

TABLE247-7589

TABLE

CIS

CO

RO

UTERC

ONS

OLE

F/X

IBMDASD

IBMDASD

IBMAS-400

IBMAS-400

IBMS-390R34

IBM3746-900

IBM3745-17A

IBM3745-L13

IBM

3745-41D

IBM3745-A11

POM

ON

AA/

C U

NIT

#8

STO

RAG

EC

ABIN

ET

DES

K

DESKSTORAGECABINET

PAC

BELL

CAB

INET

A1

3174CABINET

TABLE STORAGECABINET

STORAGECABINET

LIEBERTPDU-C

3174CABINET

3174CABINET

3174CABINET

3174CABINET

3174CABINET

POMONAA/C UNIT

#6

POMONAA/C UNIT

#7

FILECABINET

F ILECABINET

F ILECABINET

F ILECABINET

G H I J K L M N O P Q R S T U V W X Y Z AA AB AC AD AE AF AG AH AI AJ AK AL AM AN AO AP AQ AR ASC D E FBA

2

1

3

4

5

6

7

8

9

10

11

12

13

14

15

16

FIBER MDA

COPPER MDA

IDFIDF

IDF

IDF

IDF

STRUCTURED CABLING SYSTEM (ORGANIZED, REUSABLE, FLEXIBLE CABLING)

HDA

HDA

HDAMainframe

IBM 3745s

HDA

HDA

Building Automation Systems

TIA-862 addresses structured cabling for a wide variety of other systems in data centers. Common examples: Security camerasAccess controlHVAC & electrical systems Lighting

Much more common now than in the past International equivalent ISO/IEC 11801-6:2017

Grounding & Bonding ANSI/TIA-607-B:2011 Addressed bonding and

grounding in data centers Long development (2007-2011)

because of disagreement on approaches to grounding

ANSI/NECA/BICSI-607:2011 Grounding and bonding

planning and installation methods that complements TIA-607-B

International equivalent ISO/IEC 30129:2015

Larger & Container Data Centers

TIA-942-A revision added Intermediate

Distribution Area (IDA) for large data centers

Allowed >100m distance for horizontal fiber

Support for outdoor container modules

CURRENT CHALLENGES

Energy Efficiency Much more important now than in the past Addressed in a wide variety of standards

TIA-942-A added section on energy efficiency

TIA-569 revisions adopted wider range of temperatures & humidity recommended by ASHRAE

BICSI-002:2014 revision ASHRAE TC 9.9 publications (Thermal

Guidelines) Future ISO/IEC TS 22237

Physical Security ANSI/TIA-5017:2016 & Future ISO/IEC

standard Telecommunications Physical Network

Security Standard TIA standard developed in response to

request by US Government Developed in conjunction with specialist

that design secure installations for the US government and military

Three levels of security1. basic,2. tamper resistant, 3. critical security installation

Replacement of traditional 3-tier network architecture with spine-leaf architecture

3-Tier Network Architecture Spine-Leaf Switch Fabric

• Provides non-blocking, low latency switch fabric for cloud computing• Cabling for switch fabrics addressed in TIA-942-A Addendum 1• Requires more cabling and higher speed switch-to-switch cabling –

40G, 100G, 400G

Higher Speeds Need Multiple Fibers

12-fiber MPO use 8 of 12 fibers for 40G, 100G and 200G Ethernet

400GBASE-SR16 (400G Ethernet) will use 32-fiber MPO and waste no OM3/OM4 fibers

May also be used for 800GBASE-SR16 (800G Ethernet)

Faster Deployment Added recommendation to consider preterminated cabling to

reduce installation time and improve consistency and quality of terminations

Higher Densities Changed maximum 2.1 m (7 ft) maximum cabinet/rack height

from a requirement to a recommendation permitting use of taller cabinets and racks

Higher Densities - Cautions Added recommendation to consider needs for proper labeling, cable

routing, cable management, and ability to insert and remove cords without disrupting existing or adjacent connections when selecting patch panels.

BICSI-002 Data Center Design & Implementation Best Practices

Goes beyond what is provided in the cabling standards by TIA, CENELEC, and ISO/IEC Complements the national standard for telecom cabling by providing information on a wide

range of subjects not covered in the telecom standards and provide more information

BICSI Best Practices

(e.g., TDMM, BICSI-002)

Codes(e.g., NFPA 70

NEC)

Company Standards

National Standards

TIA for US & CanadaNational

StandardsCENELEC for

Europe

National StandardsISO/IEC for many other

Pages

114 Pages 503 Pages

BICSI-002 provides information on a wide range of subjects not covered in TIA-942

BICSI-002 mostly deals with aspects of data center design other than telecomBICSI-002 PagesElectrical 120

Telecommunications 84Security 44Mechanical 40Space Planning 32Maintenance 29Commissioning 28General 25Availability 23Site Selection 20Architectural 16Fire Protection 10DCIM & BAS 8Design Process 8Structural 4Outsourcing 4Multiple Data Centers 4Energy Efficiency 4

BICSI-009 Data Center Operations & Maintenance Best Practices

Companion standard to BICSI-002, but for data centers when they are in operation Another large and comprehensive standard, that also covers a wide range of subjects, but

instead related to operations and maintenance of data centers: Governance and Assessment – Compliance, Assessment Program

Operating Procedures

Security

Maintenance Procedures

Emergency Procedures

Management

EN 50600

Structure of the standard

Positioning of BICSI-002 and BICSI-009 relative to EN 50600 & ISO/IEC TS 22237

Edge Data Centers

Future Challenges Increasing consolidation of SAN and LAN traffic (higher traffic loads and

need for less delay) Higher Speeds 800G, 1TB (8 x OM5, 32 x OM4, multi-pair single-mode?) Edge data centers (computer rooms in new environments) Smart cities, IoT (Internet of Things), smart cars, streaming replacing TV,

and other applications increasing processing, storage, and bandwidth demands of the data center

IoT and single-pair cabling bringing more devices into the structured cabling of the data center

Questions?Jonathan JewPresident J&M Consultants, Inc.Website: www.j-and-m.comEmail: jew@j-and-m.com

Secretary TIA TR-42Vice-Chair TIA TR-42.3 pathways & spacesCo-chair BICSI data center subcommitteeVice-chair USTAG to ISO/IEC JTC 1 SC 25 WG 3Editor ISO/IEC TR 14763-2-1 telecom admin identifiersSME BICSI TI&M Telecommunications Administration

Data Center Cabling Challenges Past, Present & FutureMy Past ….Data Center and Telecom Cabling StandardsIn the beginning … Large Proprietary Point-to-Point CablingIEEE 802 LAN StandardsTIA-568 Cabling StandardsTIA-606 Administration StandardImpact of Documentation, Labeling, & Equipment RefreshTIA-942 Telecom Infrastructure Standard for Data CentersPoint-to-point unstructured cablingSlide Number 12Structured cabling – faster deployment, easier to manage & troubleshootBuilding Automation SystemsGrounding & BondingLarger & Container Data CentersCURRENT CHALLENGESEnergy EfficiencyPhysical SecurityReplacement of traditional 3-tier network architecture with spine-leaf architectureHigher Speeds Need Multiple FibersFaster DeploymentHigher DensitiesHigher Densities - CautionsBICSI-002 Data Center Design & Implementation Best PracticesSlide Number 26BICSI-002 mostly deals with aspects of data center design other than telecomBICSI-009 Data Center Operations & Maintenance Best Practices Slide Number 29Positioning of BICSI-002 and BICSI-009 relative to EN 50600 & ISO/IEC TS 22237 Edge Data CentersFuture ChallengesQuestions?