Designing Hadoop for the Enterprise Data Center

  • View
    6.500

  • Download
    1

Embed Size (px)

DESCRIPTION

Strata/Hadoop World 2012 with Jacob Rapp, Cisco & Eric Sammer, Cloudera

Text of Designing Hadoop for the Enterprise Data Center

  • 1. Designing Hadoop for the Enterprise DataCenter Jacob Rapp, Cisco Eric Sammer, Cloudera
  • 2. AgendaHadoop Considerations Traffic Types Job Patterns Network Considerations ComputeIntegration Co-exist with current Data Center infrastructureMulti-tenancy Remove the Silo clusters 2
  • 3. Data in the Enterprise Data Lives in a confined zone of enterprise repository Long Lived, Regulatory and Compliance Call Sales ERP Doc Recor Doc Driven Cente Pipeli Modul Mgmt ds Mgmt r ne eA A Mgmt B Heterogeneous Data Life Cycle Data ERP Soc Office Video Many Data Models Servic Media Modul Apps Conf Collab e eB Diverse data Structured and Unstructured Produc Diverse data sources - Subscriber based Customer DB t Catal og VOIP Exec Report (Oracle/SAP) Catalo Data s Diverse workload from many g sources/groups/process/technology Virtualized and non-virtualized with mostly SAN/NAS base Scaling & Integration Dynamics are different Data Warehousing(structured) with diverse repository + Unstructured Data Few hundred to thousand nodes, few PB Integration, Policy & Security Challenges Each Apps/Group/Technology limited in data generation Consumption Servicing confined domains 3
  • 4. Enterprise Data Center InfrastructureWAN Edge Layer FC FC SAN A SAN B Nexus 7000 Layer 3 MDS 9500 10 GE Core Layer 2 - 1GE SAN Layer 2 - 10GE DirectorCore Layer 10 GE DCB(LAN & SAN) 10 GE FCoE/DCB 4/8 Gb FC Nexus 7000 10 GE Aggr vPC+ L3 FabricPathAggregation & Services L2 Layer Network Services FC FC SAN Access SAN A B Layer Nexus SAN Edge 5500 MDS 9200 / FCoE 9100 B22 FEX Nexus 5500 10GE CBS 31xx Nexus 7000 Nexus 5500 FCoE UCS FCoE HP Bare Metal Nexus 2148TP-E Blade switch Nexus 2232 Nexus 3000 End-of-Row Blade 1G Nexus 3000 Bare Metal Top-of-Rack Top-of-Rack C- Top-of-Rack 10G class 1 GbE Server Access & 4/8Gb FC via dual HBA (SAN A // SAN B) 10Gb DCB / FCoE Server Access or 10 GbE Server Access & 4/8Gb FC via dual HBA (SAN A // SAN B) 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 4
  • 5. Hadoop Cluster Design &Network Architecture 5
  • 6. Validated 96 Node HadoopCluster Nexus 7000 Nexus 7000 Nexus 5548