State of the Data Universe

State of the Data UniverseWhat the best are doing and why your data is your

future

Kenny GormanChief Technologist; Data at Rackspace

Co-Founder, ObjectRocket

@rackspace @kennygorman

Big DataWTF?

Big DataIt’s just data. It’s really important, and there

is probably lots of it, so maybe we call it:

‘really important data’

Focus on using data to be competitive

Forget how big it is or isn’t

Data is your competitive advantage

We're entering a new world in which data may be more important than software.

- Tim O'Reilly

The changing landscape

Today

DSS

BI

Big Data ????Analytics

????NoSQL

RDBMS NewSQL

● Operational Stores(PostgreSQL,MongoDB)

● Big Data (Hadoop)

● Streaming (Kafka)

Level Set

Value of the data?

1. Business Intelligence

2. Product Improvement

3. Operationalization

System Types

● Beyond traditional BI

● Capture important data, instrument everything important

● Pick your systems wisely, the right tool for the job

● Build automation & product based on data.

● Augment and extend existing systems.

Succeeding

Examples

Rush Hour Rewards is a service that helps you earn money back from your energy company by using less energy when everyone else is using more.

Using data to be more efficient with finite product availability, optimizing overall cost of goods, and thus increasing profitability

Nest

Nest

How it works(I think)

Thermostat -> API -> DB <- API <- Austin Energy

Austin Energy -> API -> Thermostat

{ "hvac_ac_state": "False",

"heat_pump_comp_threshold": "-31.5", "fan_cooling_enabled": "True", "leaf_away_high": "26.111",

"compressor_lockout_timeout": "0", "gear_threshold_low": "0.0",

"lower_safety_temp_enabled": "True", "postal_code": "78730", "learning_mode": "True", "country_code": "US",

"heat_x3_source": "lp", "fan_timer_duration": "900",

"backplate_serial_number": "02BA03AC031406Y4", "hvac_wires": "Heat,Cool,Fan,Rc,Star",

"humidifier_type": "unknown", "target_change_pending": "False",

"sunlight_correction_active": "False", "logging_priority": "informational",

"temperature_lock": "False", "dual_fuel_breakpoint_override": "none",

"has_x3_heat": "False", "alt_heat_x2_delivery": "forced-air",

"maint_band_lower": "0.56", "auto_away_learning": "ready",

"device_locale": "en_US", "learning_time": "1794",

"timestamp": "1405831101750", "time_to_target_training": "ready",

"has_fan": "True", "auto_dehum_state": "False",

"star_type": "unknown", "backplate_model": "Backplate-2.8",

"heat_x2_source": "lp", "aux_heat_source": "electric", "filter_changed_date": "0",

"equipment_type": "electric", "dehumidifier_orientation_selected": "unknown",

"forced_air": "True", "name": "Master",

"can_cool": "True", "aux_lockout_leaf": "10.0",

"filter_reminder_level": "0", "humidifier_state": "False",

"error_code": "", "leaf_threshold_cool": "23.888",

"has_x2_cool": "False", "hvac_pins": "W1,Y1,Rc,G,Star",

"creation_time": "1400446808068", "heat_pump_comp_threshold_enabled": "False",

"pin_star_description": "none", "compressor_lockout_enabled": "False", "learning_days_completed_cool": "60",

"away_temperature_low_enabled": "True", "note_codes": "[]",

"leaf_threshold_heat": "1000.0", "y2_type": "unknown",

"cooling_source": "electric", "leaf": "False",

"auto_dehum_enabled": "False", "alt_heat_x2_source": "gas",

"hvac_aux_heater_state": "False", "learning_days_completed_heat": "2",

"has_humidifier": "False", "gear_threshold_high": "0.0",

"current_schedule_mode": "COOL", "target_temperature_type": "cool",

"backplate_bsl_info": "BSL", "version": "-978861903",

"fan_cooling_readiness": "ready", "battery_level": "3.864",

"temperature_lock_high_temp": "22.222", "humidity_control_lockout_end_time": "0",

"target_temperature_high": "24.0", "ob_persistence": "True",

"hvac_safety_shutoff_active": "False", "schedule_learning_reset": "False",

"pin_y2_description": "none", "is_on_stand": "False",

"emer_heat_source": "electric", "filter_reminder_enabled": "False", "compressor_lockout_leaf": "-17.8", "aux_heat_delivery": "forced-air", "away_temperature_high": "24.444",

"fan_duty_start_time": "0", "learning_days_completed_range": "0", "target_humidity_enabled": "False",

"switch_system_off": "False", "sunlight_correction_ready": "True",

"sunlight_correction_enabled": "True", "has_emer_heat": "False",

"safety_temp_activating_hvac": "False", "has_dual_fuel": "False",

"heatpump_setback_active": "False", "has_heat_pump": "False",

"fan_control_state": "False", "model_version": "Display-2.8",

"has_aux_heat": "False", "current_version": "4.2.4",

"away_temperature_high_enabled": "False", "can_heat": "True",

"alt_heat_delivery": "forced-air", "current_humidity": "41", "target_humidity": "35.0",

"upper_safety_temp": "35.0", "heater_delivery": "forced-air",

"where_id": "00000000-0000-0000-0000-000100000006", "backplate_mono_version": "4.0.21",

"has_fossil_fuel": "True", "mac_address": "18b4302ae97f",

"serial_number": "02AA01AC041402HT", "type": "TBD",

"lower_safety_temp": "4.444", "hvac_heater_state": "False",

"humidity_control_lockout_start_time": "0", "fan_mode": "auto",

"filter_changed_set_date": "0", "range_enable": "True",

"heatpump_savings": "off", "radiant_control_enabled": "False", "temperature_lock_low_temp": "20.0",

"pin_ob_description": "none", "auto_away_reset": "False",

"humidity_control_lockout_enabled": "False", "fan_duty_cycle": "3600", "heatpump_ready": "False",

"preconditioning_enabled": "False", "hvac_fan_state": "True",

"preconditioning_ready": "True", "target_time_confidence": "0.0",

"local_ip": "192.168.1.144", "pin_w1_description": "heat",

"current_temperature": "22.43", "has_air_filter": "True",

"cooling_x2_source": "electric", "hvac_alt_heat_state": "False",

"heat_pump_aux_threshold": "10.0", "rssi": "58.0",

"fan_timer_timeout": "0", "has_alt_heat": "False",

"leaf_schedule_delta": "1.11", "backplate_bsl_version": "2.1", "user_brightness": "medium",

"preconditioning_active": "False", "pin_w2aux_description": "none", "pin_rc_description": "power", "has_dehumidifier": "False", "maint_band_upper": "0.56",

"target_temperature": "22.778", "leaf_learning": "ready",

"emer_heat_delivery": "forced-air", "pin_y1_description": "cool", "capability_level": "4.0",

"pin_rh_description": "none", "available_locales": "en_US,fr_CA,es_US,en_GB",

"dehumidifier_state": "False", "hvac_emer_heat_state": "False", "dehumidifier_type": "unknown",

"nlclient_state": "", "hvac_heat_x2_state": "False",

"upper_safety_temp_enabled": "False", "learning_state": "slow",

"hvac_heat_x3_state": "False", "hvac_cool_x2_state": "False", "fan_cooling_state": "True", "fan_duty_end_time": "0",

"auto_away": "0", "alt_heat_source": "gas",

"heat_link_connection": "0", "temperature_lock_pin_hash": "", "cooling_delivery": "unknown",

"heat_pump_aux_threshold_enabled": "True", "leaf_away_low": "16.67",

"heat_x3_delivery": "forced-air", "ob_orientation": "O", "touched_by": "{}",

"temperature_scale": "F", "emer_heat_enable": "False",

"backplate_mono_info": "TFE (BP_D2) 4.0.21 (root@bamboo) 2014-05-02 16:54:17", "auto_away_enable": "True", "pin_g_description": "fan",

"click_sound": "on", "hvac_alt_heat_x2_state": "False", "target_temperature_low": "20.0",

"has_x2_heat": "False", "away_temperature_low": "10.0",

"time_to_target": "0", "heat_x2_delivery": "forced-air", "cooling_x2_delivery": "unknown", "dual_fuel_breakpoint": "-1.0",

"_id": "ObjectId(53cb494e2239c261ac83dfe0)", "heater_source": "lp",

"pin_c_description": "none", "has_x2_alt_heat": "False"

}

Results

Why this is the future● It’s may be big data, but maybe not.

● Multiple entities, communicating via API

● Multiple layers of data analytics

● Very competitive

Rackspace Cloud Insights

How do I cut down the number of false positives in my monitoring solution?

Rackspace Cloud Insights● Pattern & anomaly detection algorithms

● Variation of Bollinger Bands algorithm

● First order differencing

● Belief network & Vector Similarity

Rackspace Cloud Insights

Rackspace Cloud Insights

metrics store -> analysis componentry -> alerting engine

https://developer.rackspace.com/blog/rackspace-cloud-intelligence-insights-in-monitoring/

Why this is the future● Answers a very simple, but powerful use

case

● Heterogeneous solution

● Very competitive

● Your brilliant data scientist spends all his/her time wrangling data not producing insights

● Technology choices gone wrong

● Capture lots of data, but never analyze it

● Waiting too long to start capturing data

Data Anti-Patterns

Data is a precious thing and will last longer than the systems themselves.

- Tim Berners-Lee

Contact

@kennygorman@rackspacekenny.gorman@rackspace.com