Copyright © 2008 SAS Institute Inc. All rights reserved. SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks

Copyright © 2008 SAS Institute Inc. All rights reserved. SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. ® indicates USA registration.

Cheryl Doninger Nancy RauschR&D Director, SAS Senior Software Mgr, SAS

Data Integration in a Data Integration in a Grid-Enabled Grid-Enabled EnvironmentEnvironment

Copyright © 2008, SAS Institute Inc. All rights reserved. SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. ® indicates USA registration.

30 Years Ago - the Mainframe


Exploiting multiple processors in a machine


Grid goes beyond a single machineSAS Grid Manager


SAS Grid Manager Key CapabilitiesSAS Grid Manager

Distributed Enterprise Scheduling

Workload Balancing

Parallelized Workload Balancing

Distribute parallelized SAS workloads to a shared pool of resources.

Distribute workloads to a shared pool of resources.

Distribute jobs within workflows to a shared pool of resources.

Optimize the Efficiency and Utilization of Computing Resources


What Products Can Leverage SAS Grid Manager?SAS Grid Manager


Workload BalancingParallelized Workload

Balancing

SAS Data Integration Studio

SAS Enterprise Miner

SAS Risk Dimensions*

Any SAS program*

SAS Stored Processes**

*(with modification)**(with limitations)


SAS Enterprise Guide*

SAS Workspace Server

Any SAS program*


*(with wrapper)**(with limitations)


SAS Web Report Studio

SAS Marketing Automation

SAS Marketing Optimization

Any SAS program*

*(with modification)


SAS Code Importer


Once Imported ...

http://support.sas.com/documentation/onlinedoc/gridmgr/index.html


SAS Data Integration Studio – Distributed Enterprise Scheduling


SAS Data Integration Studio – Multi-User Workload Balancing …

PUBLIC SECTOR

MANUFACTURINGFINANCIAL

LIFE SCIENCES


Data Integration Studio on a Grid: Loops and Iterations

Example: A simple job

Specific physical tables referenced

Specific transform logic


Repetition can be helpful

Processing data in multiple pieces

Same process over several data sets

Examples:• Same process every hour

• Same process for multiple stores

• Same process for every state


How to do repetitive things?

Here is one way

Copy, Paste

Edit in new job

Problem: Multiple maintenance


Doing this more automatically

Use Looping

• Loop

• Loop End


How to do iteration

Loop input:

• list of items to repeat over

Loop body:

• one or more jobs and transforms to run repeatedly

Loop output: status table (optional)

• Can be input into next loop


How to loop

Sequential• One SAS session runs

them all

Parallel• Connect licensed

− Parallel on samemachine SMP

• Grid Manager licensed

− Parallel on a grid


How to loop in parallel – Lots of options

1 per CPU• Don’t overload machine

Specified number• Help prevent overload

• Can double up per CPU

Run all• Let ‘er rip!


Controlling iteration with parameters

By default, tables are specific physical locations

Many things in SAS can accept macro variables

Parameters are macro-enabled ETL objects

Data Integration Studio provides user interface

Input values can be mapped to parameters


Creating Parameters

Parameter name

Macro variable• &StateParm

Default value• Used in many ETL/S

activities− Running a test job− Viewing data


Parameters on objects

Property tab: add parameters for that object

Jobs can import them• From referenced tables

• From included nested objects

Loop transform will use them

Can supply a default for testing


Some good Examples of Parameters

In a table name• RETAIL&StateParm

In a filepath

In library path

ODS Titles

Mapping

SQL Query

…Anywhere you want…


Real Example: Start with 1 Retail Store

1,000,000 orders

1 year =

80 MB data


Scale up the data

10,000 stores

52 billion orders

5 years =

4.2 terabytes of data


Run Jobs in Parallel by Looping

Loop transform1 Atlanta Store

2 Chicago Store

3 Miami Store

4 …


Substitute Variables

1 Week1

2 Week2

3 Week3

4 …

Add parameter to Table:

Name = &week

1 OutWeek1

2 OutWeek2

3 OutWeek3

4 …


Name = Out&week

Process LoadInput

Existing Job

Output


The Results were very good …

3.22 terabytes per hour

50 GB / minute

~1 GB / sec


Grid Partitions

Data Integration Studio

…

n

Enterprise Miner

EM grid

DI grid

Base, Connect,

Base, Connect,

Base, Connect,

SAS Grid Mgr

SASServers

Connect Client


Grid Partitions

Restart sessions

Log directory

Error handling• Abort all remaining

• Abort only current

• Continue on error

• …others

Useful Loop Transform Options


Another Case Study: Census dataRunning sequentially

Data from 50 states

Running on one computer at a time

About 580 minutes

(just under 10 hours)


Running in parallel

Running on six computers

About 108 minutes

(under 2 hours)


Adding more computers

Across nine computers

caalarazcoct

dcdenygahi

iaidil tx

inks kyfloh

la mamdme pa

mimnmoms mt

ncnenh njnm

nv okorri scsd tn

ut vavtwawi

wvndwyak

0 10 20 30 40 50 60 70 80

Minutes (showing start time and duration of each job)

Census Data - Parallel Run 13 - Execution Profile(ETL running 1 job at a time on 9 blades, 1 slot/blade)

~ 77 minutes


Did we keep the computers busy?

In this case, we really did

Running 6 jobs at a time on 6 processors


Additional Case Studies

Unstructured Data

…See the paper for more examples

Using Grid harnesses the power of your enterprise


Questions and Want to know more? Achieving High Availability in a SAS® Grid Environment, Paper

001-2009

What's New in SAS® Data Integration Studio 4.2, Paper 093-2009

For Base SAS® Users: Welcome to SAS® Data Integration!, Paper 092-2009

Cross Validation and Learning Curve Model Comparison with JMP® Genomics and Grid Computing, Paper 286-2009

ISO’s Evolution to BI on the Grid: A Customer Perspective, Mon, 5:30 PM, Maryland 3; Paper 269-2009

Going from Good to Great: The Value of an Analytic Grid Platform at ISO, Tues 11:00 PM, National Harbor 12; Presentation only

The University of Phoenix Wins Big with SAS® Grid , Tues 11:00 PM, National Harbor 5; Presentation only




Supply a Default Value for Testing

Default value: Week1


Existing Job

Extract LoadInput

Existing Job

Output


Reuse existing job and run in parallel

1 Portugal

2 France

3 Spain

4 …

Existing Job

Existing Job

Existing Job

Existing Job

Existing Job

…


Iteration (Looping) in Parallel

1 Portugal

2 France

3 Spain

4 …


Name = &Country

1 OutPortugal

2 OutFrance

3 OutSpain

4 …


Name = Out&Country

Extract LoadInput

Existing Job

Output


What products can leverage SAS Grid Manager? SAS Grid Manager


Multi-User Workload Balancing

Parallel Workload Balancing

Optimize the Efficiency and Utilization of Computing Resources


SAS Enterprise Miner

SAS Risk Dimensions*

Any SAS program*




Any SAS program*

SAS Enterprise Guide*

SAS Workspace Server




SAS Web Report Studio

SAS Marketing Automation

SAS Marketing Optimization

Any SAS program *

*(with modification)

Documents

Copyright © 2008 SAS Institute Inc. All rights reserved. SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks