Upload
tranminh
View
215
Download
0
Embed Size (px)
Citation preview
#TDPARTNERS16 GEORGIA WORLD CONGRESS CENTER
Aster. Now. Future. Why.
Michael McIntireCTO Teradata Labs, Aster
Who is that McIntire guy anyway…
• Extreme Scale MPP platforms & complex data systems
• The “Seven year stretch…”• Seven years: Geographic Information Systems… • Seven years: Teradata in the 90’s – DB Architect… • Seven years: independent EDW consultant…• Seven years: eBay EDW Chief Architect
• And some between time... Yahoo, Sears, Prime, EDS...
Roles… Market…
Role One: Ideas – Business Process
• Algorithm and Knowledge Finding• Business Process – Growth Objective • Discover and Prove • New: Classic and Citizen Data Scientists
• Speed of Ideation• Dynamic Connectivity• Broad analytic capability• Scalable performance
DESIGN PLAN BUILD TEST
REVIEW
DRIVE
EVALUATE
Role Two: Operationalization – IT Process
• Given a Known Hypothesis…
• IT - “Fitting things together”• Generalized Architecture • Secure, Repeatable, predictable
• Platform for Production• IT Process – Control for Cost• Architecture Adherence• Enterprise Features and Resiliency• Fed by the Business process…
ProductionPlatform
Platform view of the market…
• Expanding Compute Engines• Narrower Focus• More capable point solutions
• Diverging Storage and Compute• APIs are the Enabler
• Consolidating Storage Engines• Rise of “Good enough”
• Former Platforms as Engines...• “Pluggable” Infrastructures
ComputeStorage
HDFS LocalCEPH
Ecosystem is the Platform
Connecting Multiple Platforms at Runtime
• Aster Compute within Hadoop Ecosystem• First Class Citizen – YARN Resource mgmt• Native Read/Write HDFS Storage
• MPP Interoperability with External Engines• Native Aster and Spark integration
• Teradata + Aster + Hadoop
Accellerate Business Decision making with Platform Interoperability
Engine Workflow Integration• Every Analytic Engine will have one
• Just like visualization…• First Generation: AppCenter
• Workflows will Mix Paradigms • Set processing + Procedural
So What’s New: • Command Language Implementation
• Server Side implementation• Visual Tools Layered on Commands• Data “Cooking” tools already there…
• Tech issue: Exposing Logic inside a set statement across proc bounds.
• Predicate Search for example• Optimization across loop/branch
constructs
What kind of play? Play Hard or Money? Creative money?
PLAY WORK & PLAY WORK
Snake action game First Shooter Be an Actor What kind of Work?
FACEBOOK GAMING EXTREME JUST NOTES YESNO
Why not get An Opad?
NO YES NO YES
NO YESRIGHT
FALSEGet a
Success
NOTEPAD
Get aSucces
s?
NOTEPADBROKEN
RIGHT FALSE
Snakeaction game
Play Hard or Money?
RIGHT FALSE
NO YES
RIGHT FALSEGettingA success
OutcomePlayHard
orMoney?
NO YESNOTEPAD
Inputs
Aster Next
Objective - Aster in the Cloud
• Aster 6.2x on Appliance• Aster on Hadoop (AsterX 7.0)
• Aster 6.20 on AWS• Aster 7.x on Cloud
• Managed, Public
• Not likely to see: • Aster on Hadoop on the Cloud…
Compute
StorageHDFS LocalCEPH
AsterX Evolution
• Aster Execution Engine • Aster is a Compute Engine• Spill to disk temp storage only
• Non-Persistant Storage • Access via Connectors
• QueryGrid2
Compute
StorageHDFS LocalCEPH
6.20 Worker – Aster Managed Storage
Worker Node
ASTER - vWorkerASTER - vWorkervWorkerMany per Node
RelationalEngine
Map ReduceEngine
Node Local Storage – OS Managed
GraphEngine
Compression Replication AsterAsterAsterLocal Storage
Aster
EdgeNodeQueen
Exec
User
Cluster Services –1 per node
6.50 Worker – 100% HDFS Storage
Aster
HDFS Cluster
EdgeNodeQueen
Exec
User
Hadoop
NameNode
YARN
Distributed File System (HDFS on HDP, CDH)
HadoopWorker Node
ASTER - vWorkerASTER - vWorkervWorker m/node
RelationalEngine
Map ReduceEngine
Cluster Wide Storage: HDFS:/aster/vworkerX*Y
GraphEngine
Compression Replication Security Management
Cluster Wide Storage Interface
Cluster Services –1 per node
AsterX 7.0 Worker – Local Temp
HadoopWorker Node
ASTER - vWorkerASTER - vWorkervWorker m/node
Cluster Services –1 per node
RelationalEngine
Map ReduceEngine
Distributed Persistant Data System Hive, Teradata,
GraphEngine
Aster
HDFS Cluster
EdgeNodeQueen
Exec
User
Hadoop
NameNode
YARN
NODE LOCAL Storage: /aster/vworkerX*Y
Aster 7.x Architecture
AsterX 7.0 Cluster Architecture• Internally: Daemon based implementation
• Always on - not per job instantiation
• vWorker Deployment: Cluster Subset• vWorker count is Static per Instance • vWorkers can be moved• Expand / contract Hadoop Cluster
• without Aster intervention• Architected as a SUBSET
• Queen edge node (required)• Security and Connectivity (++ eliminate
bridge)
• Libraries on all nodes • For Simplicity and Latency reasons
EdgeNode
App AQueen
User
HadoopServices
NameNode
Aster B
HDFSAster A
HDFS Map/Reduce
HDFS
Aster A
HDFS
Hive
Aster B
EdgeNode
App BQueen
User
WorkerNodes
Head Nodes
YARN Managed Resource
• Full Hadoop Services Integration
• Injects Third Party Management
• Reverse order Worker setup/teardown
• Aster Cluster still manages “State”
• Consul implementation
Yarn ManagedEdge Node
Hadoop Services
YARN ManagedWorker Node
ASTERvWorkerHDFS
Aster
HDFS
ASTERQueen Aster Yarn
Server YARN
User
Ambari Zookeeper
YarnClient
1
2
34
AsterX 7.0 – Consul State Management
• Consul: State management, configurations
• Simple, always on key-value store• Similar to ZooKeeper (Dir/Key structure)
• Aster 7.0 use:• Common, resilient store port mapping• Future use dynamic mapping of ports
• Dynamic worker movement…
• Consul is required• AX7.0 is a Private Implementation
• Future use of existing Consul possible• If not available – Aster will not come up
Queen
User
Aster Aster
Temp Temp
Consul
AsterX 7.0 Cluster Configuration
• Subset of nodes: explicitly or system decides
• Exact # nodes will fit node capacity • i.e if the nodes are powerful there will be fewer
nodes used• Alternate maxusage yarn parm for temp/io
heavy apps• Equivalent of “Prepared” state, still needs
“activate”
• Port Configuration –• startup time: port conflicts can be resolved• Re-address when new Cluster SW is
installed
• No add/remove node functionality• Stand up another cluster… Point to the data• No “data migration”...
Queen
User
Aster Aster
Temp Temp
AsterX 7.0 Cluster Startup
• Install, startup - separate steps• Install libraries, basic directories
• Startup plumbs all connections• Setup vWorkers• Connect Queen and Workers
• Shutdown - cleans up the workers• Temp data removed• Reuse temp : Future Optimization Case
• All via Aster Yarn Client Commands• Equivalent to Aster “Activate”
Queen
User
Aster Aster
Temp Temp
Consul
AsterX 7.0 Worker – Local Temp Only
• Persistence in HDFS • Hive, hCat…
• Access: Connectors + QueryGrid
• Read at script Start / Write at script End
• Objects managed by user• Same semantics as a Database• Persist for Duration of Cluster
• No Replication & Compression• Redistribution remains
Hadoop NodeHadoop NodeHadoop NodeAster Connectors – Query Grid
ASTER - vWorkerASTER - vWorkervWorker m/nodeSQL
Engine
M/RAPI
Engine
GraphEngine
HDFS Cluster
EdgeNodeQuee
nExec
User
NODE LOCAL Storage
Hadoop
NameNode
YARN
AsterX Local
Storage
Advanced Analytics Enabled by SQL (for Data/Business Analysts)
Once you know how to use on Aster SQL command you have learned how to use them all!
CREATE TABLE complaints_nb_model(PARTITION KEY(token)) AS
SELECT token, SUM(crash) AS crash, SUM(no_crash) AS no_crash
FROM NaiveBayesText (
ON complaintsTEXT_COLUMN ('text_data')CATEGORY_COLUMN ('category')CATEGORIES ('crash', 'no_crash') )
GROUP BY token;
ANSI SQL Statement
SQL MR Statement
Data Source
SQL-MR Predicates
AsterX 7.0 Storage
• Examples: Analytic Temp Tables and Hive Perm Tables
TEMPStorage
PermStorage
Queen
AX AX AXSQL
Temp Temp
Distributed Storage: Hive, hCat
LocalSpill to
disk
Foreign Server Read/Write w SQL-H
HiveTable
Weblog
TEMPStorage
PermStorage
AsterX Storage
• Before AX is running• Hive tables: Hive_t1, Hive_t2, Hive_t3• Flat Files: weblog.txt
Queen
AX AX AXSQL
Temp Temp
Distributed Storage: Hive, hCat
LocalSpill to
disk
Foreign Server Read/Write w SQL-H
HiveT1
HiveT2
HiveT3
Weblog
TEMPStorage
PermStorage
AsterX Storage
• CTAS – analytic: Aster_analytic_t1• CTAS – Temp: Aster_session_temp_t2
Queen
AX AX AXSQL
Temp Temp
Distributed Storage: Hive, hCat
LocalSpill to
disk
Foreign Server Read/Write w SQL-H
Session Lifetime
Uptime Lifetime
HiveT1
HiveT2
HiveT3
Weblog
T1 T2
TEMPStorage
PermStorage
AsterX Storage
• SQL query – phase temp tables• Temp_phase_1 (the real name would be like
_tmp_21398041237)• Temp_phase_2• Query_output
Queen
AX AX AXSQL
Temp Temp
Distributed Storage: Hive, hCat
LocalSpill to
disk
Foreign Server Read/Write w SQL-H
HiveT1
HiveT2
HiveT3
Weblog
T1 T2
TP1 TP1 QO
Query Lifetime
TEMPStorage
PermStorage
AsterX Storage
• CTAS – To Hive• Alan_dailyreport_06_24_2015
Queen
AX AX AXSQL
Temp Temp
Distributed Storage: Hive, hCat
LocalSpill to
disk
Foreign Server Read/Write w SQL-H
HiveT1
HiveT3
Weblog
T1 T2
.. AlanDR
TEMPStorage
PermStorage
AsterX Storage
• After Aster shutdown
Queen
AX AX AXSQL
Temp Temp
Distributed Storage: Hive, hCat
LocalSpill to
disk
Foreign Server Read/Write w SQL-H
HiveT1
HiveT3
Weblog
.. AlanDR
AsterX Failure & Recovery
Metadata persistance
• Admin DDL Checkpoint• Saves checkpoint file to disk• Manually done via ncli command• Restart causes checkpoint to be replayed• Checkpoint files are valid on any AsterX instance*
• Check pointed• Users, Roles, Databases, Schemas• foreign server definitions• packaged analytics models and functions• grant privileges on above
• Not Check pointed• Tables, views, constraints, indexes• R scripts installed on the server side• user-installed files and SQL/MR functions• user scripts for vacuum or daily jobs
AsterX 7.0 Failure Recovery
• Node Failure = loss of analytic tables
• Worker Node AND/or vWorker• System will allocate new node
• Conversation with YARN… • Move vworkers/Node• Come to prepared state• Activate – automatically
• ALL TEMP Data is LOST. • Vworker is treated as node failure
Edge Node
Queen User
Aster Aster
Temp Temp Temp
Edge Node
Queen User
Aster Aster
Temp Temp Temp
Before
After Restart
AsterX 7.0 Failure Recovery
• Queen Fails• Recovery is “Repair”• DDL Gen Unwind…
• If unrecoverable• Delete cluster• cluster create…
EdgeNode
App AQueen
User
HadoopServices
NameNode
Aster A
HDFSAster A
HDFS Map/Reduce
HDFS
Aster A
HDFS
Hive
Aster B
EdgeNode
App BQueen
User
WorkerNodes
Head Nodes
AsterX 7.0 Failure Recovery
• Other issues• Same behaviors, different
impact
• DDL Gen – SQL Script of the dictionary.
• State is lost – Temp data will be deleted on restart
EdgeNode
App AQueen
User
HadoopServices
NameNode
Aster A
HDFSAster A
HDFS Map/Reduce
HDFS
Aster A
HDFS
Hive
Aster B
EdgeNode
App BQueen
User
WorkerNodes
Head Nodes
AsterX 7.0 Expansion
• New Instance. Got that?
• Create New Aster Instance• Setup Foreign Server
Constructs… • Go … • Reference Existing
Persistent Data
EdgeNode
App AQueen
User
HadoopServices
NameNode
Aster A
HDFSAster A
HDFS Map/Reduce
HDFS
Aster A
HDFS
Hive
Aster B
EdgeNode
App BQueen
User
WorkerNodes
Head Nodes
AsterX 7.x Configuration Options…
Many, many more options
• Single cluster per workload… • Or... Xmas sized Cluster...
• Monthly Term licensing???
• Internal HDFS Chargeback?
• LOB specific Aster Instance • Delegation of adminstration...• Simplified CapEx / OpEx
administration
EdgeNode
App AQueen
User
HadoopServices
NameNode
Aster A
HDFSAster A
HDFS Map/Reduce
HDFS
Aster A
HDFS
Hive
Aster B
EdgeNode
App BQueen
User
WorkerNodes
Head Nodes
Aster Persistent Storage& Access - Query Grid Two
Aster 7.10 QueryGrid Two - Next Gen
• High speed TD, Presto, Hadoop connectivity• Cluster to Cluster connectivity • Point to Point model – not hub (Kafka is a Hub)
• Common Framework included in each product• Communications, State, Error Management, Data Conversion• Network Protocol, Parallelism, Distribution … and more
• Single cost implementation• Simple set of Get/Set operations specific to the implementation
• Uses full matrix communications in first release• Blocks of Tuples are distributed round robin• Full Communication Matrix• Session Data is MultiPlexed
• Multiple sessions use same communications channelAster Aster
TD TD TD
Aster Aster
TD TD TD
CurrentConnectors
QG2
Aster - Foreign Server Syntax Support
• DML syntax - external objects• Teradata’s Foreign Server Syntax
• Aster & source: Bi-directional data movement
• Load_from_Hcatalog, Teradata, etc• Load_to_Hcatalog, Teradata, etc
• Use: SEL,INS,Views and CTAS
• Query pushdown, Query time special & override of parameters also supported
• Grant & Provoke USAGE & EXECUTE privileges
CREATE FOREIGN SERVER nameUSING server(‘1.1.1.1') port('1234') …DO IMPORT WITH Load_from_XYZ USING …DO EXPORT WITH Load_to_XYZ USING …
SELECT * FROM table@foreignServer;
INSERT INTO table@foreignServerSELECT id, value FROM asterTable;
WITH FOREIGN SERVER fsAlias as (foreignServer using username('foo') password('bar') )SELECT * FROM table1@fsAlias, table2@fsAlias…;
AsterX 7.0 – Scripting Pattern Changes
• Existing Customers - Implementing persistence in AX
• Best practices in script writing• Disable Failure mode until after DDL commands• Truncate Tables (delete from all)• Create Tables inline (keeps code in one place, enables
operator to drop table and not have to change production code)• Cascading Insert/Selects/CTAS• Pour over tables for failure/locking latency
• Option of creating a cluster sized just for this workload
Aster 7.xOther Cool Stuff
AsterX 7.0 Planner Changes
• Improve plans for external table (ET) queries• External tables are the norm in AX (exception in AD)
• 7.0 Planner Hive Meta-store to get table size
?
Sales Store
? ?
Region
AD 6.50 planner view of ETs
3,000,000,000 6,000 4
Sales
StoreRegion
AX 7.0 planner view of ETs
AsterX 7.0 Planner Changes
• Planner Hive Meta-store to get base table stats• Only table rowcount and size. No columns/histograms
• Recognize small ETs and replicate as dim tables• Save on costly data repartitioning
• Improve join order optimization• Avoid early theta joins and dataflow multipliers
• Better skew avoidance• Avoid partitioning on low cardinality columns
AsterX 7.0 Multi-Tenancy
• Hadoop is an Execution Environment• Aster must conform to Hadoop’s capabilities • Hadoop supports “Sessions” and Aster
supports ”Sessions”…• Ergo – how does Aster run inside Hadoop... • Aster is a Daemon based architecture...
• Multi-Tenancy in AX7.0• Co-exist with other Hadoop Applications• Port Mapping is the largest single problem
EdgeNode
App AQueen
User
HadoopServices
NameNode
Aster A
HDFSAster A
Aster B Map/Reduce
HDFS
Aster A
HDFS
Hive
Aster B
EdgeNode
App BQueen
User
WorkerNodes
Head Nodes
Thank You
Questions/CommentsEmail:
Follow MeTwitter @
Rate This Session # with the PARTNERS Mobile App
Remember To Share Your Virtual Passes
DataOcean
598
45
Aster 7.10 – What’s NEXT
(where’s the cool graphic???)
Aster 7.10 Containers (using Docker)
• Objective: • Architecture using Containers (what processes go where) - Hadoop
• Major Impact to Startup, Distribution, Process Management
• Reality: • Extraordinarily difficult on Hadoop
• Required complete rewrite of all process management
• Theory issue problems• Process Allocation and management • Foundation of AsterX on other platforms - GCP, AWS, Azure...
• Open decision on long term Hadoop implementation
Aster 7.10 Planner Improvements
• Pushdown predicate to Hive• Automatic in 7.10 (manual in 7.0)• Reduces data movement• Utilizes store format filters
• Cuts down IO & CPU
• Increase dependence on Stats• Foreign System Get/Set Scan
Filter
…
Scan + Filter
…
Aster 7.10 Planner Improvements
• Planner pushdown sub-queries to Hive• Intelligent push down (semantics, type, size) • Minimal data movement • Better stats & data distribution utilization
ET1
T4
ET2 ET3
GB
ET1
T4
ET2 ET3
GB
NPath NPath
Aster 7.10 Aster Spark
• Utilize Spark execution framework for Aster• Aster query operator as Spark functions / scripts• Uses Spark MLlib analytics libraries • Customers write functions in Spark using…
• Uses familiar SQL/MR language framework• Support multiple Spark clusters (ex: same query)• Parallel data transfer (Sockets or HDFS)• Spark Job Monitoring
Aster 7.10 Spark Aster
• Aster table/queries use Spark Data frame API• Read Aster tables/queries in parallel
• Can cache data on disksqlContext.readAsterTable (“<table-name>”, “cache-on-disk>, …)sqlContext.readAsterUsingQuery (“<query>”, <cache-on-disk>, …)
• Write Data frames Tables in parallel (overwrite / append mode)<dataframe>.writeToAsterTable(“table-name”, <mode>, …)
Existing Framework (Analytic flow)
Batch-mode Processing
ASTER FRAMEWORK
Analytics(Model Builder)
Prediction Response
PredictionRequests
AppropriateAction
Analytics(Predictor)
Queries(Test Data)
TrainingData
Model
Score
RealTime Platform -Asynchronousfeedback between the two frameworks.
Aster Platform - Prediction analytics isolated from training (modeling)
Proposal: Split the processing… Real-time Scoring
Analytics(Model Builder) Prediction
Response
PredictionRequests
AppropriateAction
Analytics(Predictor)
Queries(Test Data)
TrainingData
Model
Score
Generates AML File from Model Table
Aster Model Language GeneratorReal-time Scoring
Analytics(Model Builder) Prediction
Response
PredictionRequests
AppropriateAction
Analytics(Predictor)
Queries(Test Data)
TrainingData
Model
Score
AMLGenerator
DRIVERFUNCTION
Your Real Time FrameworkAML File
Scorer Execution Flow
Prediction Response
Transport as
Java JAR file
AppropriateAction
Score
Request
Response
Configurator
Model TypeModel DefinitionModel DataRequest ParametersRequest Definition
PredictionRequests