Rackspace Analytical Compute Grid (ACG)

April 12, 2023

Analytical Compute Grid (ACG)

Elastic “Big Data” Infrastructure

by Natasha Gajic

Big Data on Open Cloud

2RACKSPACE® HOSTING | WWW.RACKSPACE.COM

Rackspace’s EBI Environment

Current EnvironmentWindows and Linux

operating systemsOracle and Microsoft

databases solutionsMicrosoft and Oracle

replication technologySSISInformaticaDedicated serversRapid data set growth

“Big Data” ProblemCost of purchasing

additional licensesTime required to set up

new hardwareIncreased demand for DBA

resourcesSystem performanceSystem scalabilityCapacity


Analytical Compute Grid (ACG) Features

•Host ever growing set of data•Quick data collection and retrieval•Rapid scalability•Ease of maintenance•Provide standard data access API


Analytical Compute Grid (ACG) Features

•Ability to provide variety of storage types:

• Columnar

• Relational

• HDFS

•Enable users to select optimal storage type for information collected

•Leverage Rackspace® Private Cloud powered by OpenStack® and open source technology


Analytical Compute Grid (ACG) Quality Attributes


ACG on Rackspace® Private Cloud powered by OpenStack®

High Level Architecture





Image



Database Engine Selection

Columnar Cassandra

Relational PostgreSQL

HDFS Hadoop



Node



Node



Node



Node



Controller



Controller



Controller



API



Indexing Structure



Indexing Structure



Indexing Structure

What is ACG Indexing Structure?

• System entry point

• Set of pointers ultimately addressing database entities



Indexing Structure

What is ACG Indexing Structure?

• System entry point• Set of pointers ultimately addressing database entities

Where is Indexing Structure Located?

• It is a part of ACG so it resides on Open Cloud• ACG Controller manages Indexing Structure



Indexing Structure

What ACG Indexing Structure Enables?

• Splitting of large data sets across many instances• Query parallelization• Controlled data store size• Optimal data store configuration• Uniform access to data residing in various storage types• System scalability as it expands horizontally and vertically to address ever growing data set



Quality Attributes



Quality Attributes - Performance

Rackspace® Private Cloud powered by OpenStack®

Creates ACG node in 30 secondsCreates ACG nodes concurrentlyRe-size ACG nodes adding CPUs



Quality Attributes - Performance


Creates ACG node in 30 secondsCreates ACG nodes concurrentlyRe-size ACG nodes adding CPUs

ACG

Indexing structure and controlled data set size allow for: Quick data distribution Query parallelization



Quality Attributes – Availability


Rapidly replace failed ACG nodes



Quality Attributes – Availability


Rapidly replace failed ACG nodes

ACG

Deploys data store native availability mechanisms (replication, data distribution…)



Quality Attributes – Maintainability


Adding ACG nodes expands: Storage capacity CPU power MemoryNo DBA or system administrators activity required



Quality Attributes – Maintainability


Adding ACG nodes expands: Storage capacity CPU power RAM No DBA or system administrators activity required

ACG

Controlled data set size enables: Optimal and stable data store configuration Reducing demand for managing data store objects Stable query execution plans



Quality Attributes – Flexibility

ACG

Variety of storage types:Columnar – Cassandra : time series dataRelational – PostgreSQL : relational dataHDFS – Hadoop : un-structured data

Ability to select optimal storage type for individual use case



Quality Attributes – Usability

ACG

Standard interfaces: SQL language JDBC API ODBC

ACG Management Console

ACG Monitoring Console Loader utility implementing: Bulk Loader Insert Loader



Current State



Current State

ACG Controller

•ACG Manager•Rule Engine•Node Manager•ACG Management Console•ACG Monitoring

Columnar Implementation

•Data Store Controller•JDBC extended to work with supercolumn•Loader integrated with Informatica

Relational Implementation

•Data Store Controller•JDBC driver extended with distributed query rewrite•Loader integrated with Informatica•ODBC (In Progress)

HDFS Implementation

•Will start soon



Rackspace Use Case


• Subject:

• Complex availability calculation sourcing 3 months of monitoring data and creating 1 billion records in initial calculation


Rackspace Use Case


• Environment 1

• Data Warehouse Microsoft SQL server database• SSIS data loading• SQL server with 24 CPUs and 250GB RAM was dedicated to the initial calculation

• SQL server stored procedure performed the calculation

• Source and result are stored in traditional data warehouse structure


Rackspace Use Case


• Environment 2

• ACG running two Cassandra clusters 4 nodes each

• Informatica with Cassandra bulk loader• Each ACG node has 2CPUs and 8GB RAM• Java program running on instance with 4CPUs and 8GB RAM

• Source and result are stored in columnar structure suitable for time series data


Rackspace Use Case


• Calculation Duration

•Microsoft SQL Server lasted 5 days•ACG calculation completed in 3.5 hours

• Storage Size• Microsoft SQL server 500GB •ACG 20 GB

• Complexity of the calculation•Columnar data store is optimal for time series data. Sourcing from columnar data store resulted in relatively simple Java calculation process comparing to SQL server stored procedure


Rackspace Use Case - Result


• Selecting optimal data store for use case resulted in:

• Substantial performance improvement• Reduced storage demand•Simplified processes•Ability to process terabytes of data per day close to real-time and on-demand

•Improved trending and reporting:• enhances support capabilities

• improved Rackspace customer experience

• Significant cost reduction


Rackspace Use Case - Conclusion

40

RACKSPACE® HOSTING | 5000 WALZEM ROAD | SAN ANTONIO, TX 78218

US SALES: 1-800-961-2888 | US SUPPORT: 1-800-961-4454 | WWW.RACKSPACE.COM

RACKSPACE® HOSTING | © RACKSPACE US, INC. | RACKSPACE® AND FANATICAL SUPPORT® ARE SERVICE MARKS OF RACKSPACE US, INC. REGISTERED IN THE UNITED STATES AND OTHER COUNTRIES. | WWW.RACKSPACE.COM


ACG UI


ACG UI


ACG UI

Technology

Rackspace Analytical Compute Grid (ACG)