Embed Size (px)
Citation preview
Internet Of Things (IOT) Practice Overview
January 2016
Internet of Things Practice
2
pureIntegration has been a leading partner since the beginning of service providers offering home security as a product offering. The market has quickly grown to include home automation, health & lifestyle, connected vehicles, smart energy, and smart cities. “Smart Life” and “Internet of Things (IOT)” have been coined to describe this ecosystem, intelligently connecting our world to make life better. pureIntegration’s industry leading capability is helping to make Smart Life a reality.
Smart Life Capabilities
pureIntegration’s strength is targeted toward operators’ automation strategic road map.
1. We offer professional & managed services for home automation integration: development, certification, security penetration, testing, training, and deployments.
2. We deliver cloud, storage solutions, and analytics services for cable, telco, and IOT platform providers to build upon early success
Smart Lab
Our Smart Lab provides scalable on-ramp capacity and expertise for partners today. We offer co-location to enable joint collaboration with our team for agile and real-time results. We leverage our existing lab environments and spin up new physical and virtual environments quickly to support your on-demand needs. Our live production and development environment connections enable the highest quality solutions delivered quickly in the actual operator environments.
Systems Integration: Front-end proofs of concept “POC”, development, integration, deployment on the leading platforms and applications.Value-Driven Development & Testing: We accelerate time to market with certification, endurance, performance, interoperability, and release testing leveraging partnerships/relationships with leading developer programs and industry standards.Security Solutions: Network, device, and data security vulnerability assessment, penetration testing, & threat prevention.Data Aggregation & Correlation: Unified data collection strategy across disparate sources to provide analytics and actionable insights from network, service, and operational information
Security Practice
3
Services: Mobile and Wireless Security (Secure Mobility) IT Governance Frameworks Certification & Accreditation Security Vulnerability / Penetration Assessments FISMA & ISO Review Analysis, Mapping & Reporting Privacy Impact Review Analysis Security Engineering for Point Products Security Architecture Design Baseline Configuration to Gold Standards Risk & Threat Management Modeling Information Security Program Design / Management
Methodology: Information Technology Infrastructure Library Open Group Architectural Framework (TOGAF) Control Objectives for Information and related technology (COBIT) FISMA, FERPA, HIPAA & GLBA, Holistic Information Security Institute NIST, DIACAP, DCID/ICD & ISO, OWASP, SANS PCI DSS OMB
Tools: HP Assessment Management Platform (AMP, QAInspect, & WebInspect), CA Clarity, HP OPS Ware, Foundstone Fortify, QualysGuard, Guidance/Encase ISS, Retina, SAINT, Core Impact, Intel/McAfee, Metasploit Framework, Kali, Wireless Protocol Analyzers
Professional Certifications:
CISSP, CEH, CISA, CISM, HISP, ITIL v3, CGEIT, CFCP, GIAC, CSSLP, ISC² Associate, SSCP, ISO
Featured Case Studies & White Papers: Outbreak: Avoiding Information Security Catastrophes in Cable A Successful Track Record in Home Security and Home
Automation IOT Attack Scenarios and ZigBee Penetration Testing Security & Privacy Framework Development
pureIntegration has been a leader in the security services business for over a decade providing leadership to federal, communications, and financial services. Our certified experts apply cross-industry best practice to integrate security methodology from design through operational support.
Smart Life Center of Excellence
Our vision is to deliver Center of Excellence “COE” capabilities to accelerate innovative customer-driven solutions and manage/operationalize them successfully. pureIntegration provides mature capabilities, environments, methods, expertise, and toolsets for consistent & repeatable results.
Focus 1: Integrate Home Automation & Security Focus 3: Integrate Health & Lifestyle
Product Development
(Proofs of Concept)
Solution Development
(On-Ramp Lanes)
Integration Planning & Testing
Certification Testing & Deployments
Operational Management &
Monitoring
Managed Service Operation
Customer Experience Center & Analytic Insights
4
Focus 2: Integrate Connected Vehicles
IOT Value Proposition
5
The IOT Challenge: Rapid consumer adoption, complex integration, & limited resources• Consumers now expect all devices/appliances to be connected in the home and on the go
• Operators need to deliver added value (security, energy savings, health management, mobile convenience)
• New device integration is slow and complex in the current model (3rd Party WiFi, iControl, gateways, etc.)
• Multiple platform & connectivity standards across automation devices
• Many technical and quality issues placing strain on operations (field, support, environments, testing, etc.)
The Need: Go to market quicker with a higher level of quality and profitability• Rapid solution integration between major platforms to launch new services
• Proofs of concept to help accelerate product development
• Certification testing and baselining of new device entry criteria
• Rapid environment capacity for endurance/performance, functional, & release testing
• High quality test automation, content management, data security, analytics, consumer experience & operational monitoring
• Reduce truck rolls and support costs due to product and equipment issues
Value Proposition: Smart Life Lab & Center of Excellence
• Accelerate new platform/device integration, and proofs of concept – work in our lab 6 months in advance• Accelerate certification & on-boarding of vendor solutions – we work in Dev, QA, Prod environments• Harden environments with enhanced security, endurance, & performance testing – onsite or nearshore• Provide managed environments, operations, monitoring, & support – we are onsite every day
IOT Impact
pureIntegration gets results and enables operators to on-board new devices quickly. Once a device hits the roadmap we adopt it into our labs, realizing a 6-9 month jump-start for operators. We conduct security testing at a level that few can achieve, and shorten market deployments.
6
IOT Labs
Accelerate on-boarding 6-9 months
Certification Testing Security Penetration Testing
Validation Group # Test ExpectedTool
Required
Device #1
Visonic Water Sensor
MCT-550 SMA
Device #2
3 Series Water Sensor
3315-G
Device #3
SMC Water Sensor
SMCWA10-ZGeneral Packaging 2.1.1 Validate installation instructions are easy to understand
and clearly written
Expected: paper based instructions should be included at least.
Online instructions and help are a bonus. Instructions should be
legible and should be clear in direction and context.
- Connecting Wires was a
challenge. Pairing instructions
not 100% clear and had to revisit.
FAILED PASSED
General Packaging 2.1.2 Vendor support information present. Expected: a website, phone number, email, or other method of
contact should be included.- PASSED FAILED FAILED
General Packaging 2.1.3 Validate all contents of package are present Expected: all device components should ideally be listed on the
installation instructions and of course everything should be
included in one package.
- PASSED PASSED PASSED
General Packaging 2.1.4 Validate components are securely packaged Expected: the device should have cardboard cushions or bubble
wrap if device is not durable.- PASSED PASSED PASSED
General Packaging 2.1.5 Validate battery is included Expected: batteries should be included and completely charged. - PASSED PASSED PASSED
General Packaging 2.1.5.a a. Validate battery is not actively connected to device
when shipped
Device should include a tab between the battery and contact point
to ensure no drain.- PASSED PASSED PASSED
General Packaging 2.1.5.b b. Validate battery power level is maximum when
received, if not what is the estimated power level
percentage
Expected: Battery should be between 90%-100%
-PASSED
(100%)
PASSED
(100%)
PASSED
(100%)
General Physical Installation 2.2.1 Validate ease of installation.
General Physical Installation 2.2.1.a a. How many steps are required? How many steps noted in instructions
- 5 STEPS
No instructions provided; does not
appear to require installation -
ground placement
6 STEPS
General Physical Installation 2.2.1.b b. Did it require removal of case or panel? - YES NO YES
General Physical Installation 2.2.1.c c. If yes, was the removal of case or panel simple or
difficult?
Expected: Simple or Difficult- SIMPLE N/A SIMPLE
General Pairing 2.3.1 Validate pairing compatibil ity with the system Device should share the same wireless communication protocol - ZIGBEE ZIGBEE ZIGBEE
General Pairing 2.3.2 Validate ease of connection (add a device) Expected for ZigBee: should require a single attempt once
touchscreen or hub is in activation detection mode. simply remove
strip from battery to allow it to be automatically detected.
Expected for Wi-Fi : automatically detect SSID and allow
authentication and confirmation of connectivity.
-PASSED
(SINGLE ATTEMPT)
PASSED
(SINGLE ATTEMPT)
PASSED
(SINGLE ATTEMPT)
General Pairing 2.3.2.a a. Initial operation of an activation of device Items to Capture:
- # of Steps: (standard norm is 4 steps)
- Pairing: Easy (1-3), Moderate (4-6), Difficult (7+)
- # Attempts to Pair Correctly:
-
STEPS: 6
PAIRING: MODERATE
ATTEMPTS: 1
STEPS: 6
PAIRING: EASY
ATTEMPTS: 1
STEPS: 4
PAIRING: EASY
ATTEMPTS: 1
General Pairing 2.3.2.b b. Add additional device(s), 2 or more Items to Capture:
- # of Steps:
- Pairing: Easy (1-3), Moderate (4-6), Difficult (7+)
- # Attempts to Pair Correctly:
-
STEPS: 6
PAIRING: MODERATE
ATTEMPTS: 1
STEPS: 6
PAIRING: EASY
ATTEMPTS: 2
STEPS: 4
PAIRING: EASY
ATTEMPTS: 1
General Pairing 2.3.2.c c. Did it require a complicated sequence of pressing
buttons and light sequences - YES (SEQUENCE REQ.) NO NO
General Pairing 2.3.2.d d. Did it automatically detect once battery connected? - NO (SEQUENCE REQ.) YES YES
General Pairing 2.3.2.e e. Did it require additional steps to connect to a hub or
access point?- YES (TRIPPING OF DEVICE REQ.) YES (TRIPPING OF DEVICE REQ.) YES (TRIPPING OF DEVICE REQ.)
General Pairing 2.3.2.f f. Was tripping of sensor required to complete
connection?- YES (TRIPPING OF DEVICE REQ.) YES (TRIPPING OF DEVICE REQ.) YES (TRIPPING OF DEVICE REQ.)
General Pairing 2.3.3 Validate troubleshooting. -
General Pairing 2.3.3.a a. Does it require factor re-set or reboot of the
touchscreen or hub?
Expected: simple factory re-set and re-start the activation sequence
on the touchscreen.- NO (PREMISE PASSPHRASE REQ) NO (PREMISE PASSPHRASE REQ) NO (PREMISE PASSPHRASE REQ)
General Pairing 2.3.3.b b. Did it require a manual or auto code update? - NO (PREMISE PASSPHRASE REQ) NO (PREMISE PASSPHRASE REQ) NO (PREMISE PASSPHRASE REQ)
General Pairing 2.3.3.c c. If applicable, is the factory re-set a complicated
series of pressing buttons and light sequences or was it
a one step process?
- N/A NO N/A
General Pairing 2.3.4 Validate de-pairing and re-pairing Expected for ZigBee: should require a single attempt to de-pair and
one attempt to re-pair once touchscreen or hub is in detection
mode.
-PASSED
(1 ATTEMPT)
PASSED
(1 ATTEMPT)
PASSED
(1 ATTEMPT)
General Power 2.4.1 Validate low battery Definition of low battery:
- Does the device communication or triggering fail with low battery
if not changed once notification is sent?
Good (75-99%) : device should be performed as defined on specs
Med (26-74 %), : device should be performed as defined on specs
Low ( <25? ): notification alert is sent
Battery Tester PENDING PENDING PENDING
General Power 2.4.2 Validate Dead/No Battery (<1 Day) When battery is dead for less than one day and new battery
inserted, does the device reconnect to the HUB/TS automatically?Battery Tester PENDING PENDING PENDING
General Power 2.4.3 Validate Dead/No Battery (>1 Day) When battery is dead for greater than one day and new battery
inserted, does the device reconnect to the HUB/TS automatically?Battery Tester PENDING PENDING PENDING
General Power 2.4.4 Validate performance loss of power
General Power 2.4.4.a a. Does AC power loss to TS/HUB disrupt comms
between device and TS/HUB?
Expected: No disruption of communication; battery back up will
engage.-
PASSED
(NO DISTRUPTION)
PASSED
(NO DISTRUPTION)
PASSED
(NO DISTRUPTION)
General Power 2.4.4.b b. Does AC power loss and battery power loss to
TS/HUB disrupt comms between device and TS/HUB?
Expected: Disruption of communication due to no power at TS/HUB. - PENDING PENDING PENDING
General Power 2.4.4.c c. When AC power resumes, are comms between device
and TS/HUB disrupted?
Expected: No disruption of communication, device will reconnect
automatically- PENDING PENDING PENDING
General Power 2.4.5 Validate performance loss of internet Expected: Cellular back up will kick-in and device will remain
connected- PENDING PENDING PENDING
General Power 2.4.6 Validate battery drain level – At rest / Steady State Expected: Does battery l ife last as stated in vendor specs-
Part of ongoing managed service /
Long Term Testing
Part of ongoing managed service /
Long Term Testing
Part of ongoing managed service /
Long Term Testing
General Power 2.4.7 Validate battery drain level – While communicating / non-
trigger (Check-In)
Expected: Does battery l ife last as stated in vendor specs-
Part of ongoing managed service /
Long Term Testing
Part of ongoing managed service /
Long Term Testing
Part of ongoing managed service /
Long Term Testing
General Power 2.4.8 Validate battery drain level – When triggered Expected: Does battery l ife last as stated in vendor specs-
Part of ongoing managed service /
Long Term Testing
Part of ongoing managed service /
Long Term Testing
Part of ongoing managed service /
Long Term Testing
General Power 2.4.9 Validate battery temperature Expected: Does device work as expected within battery temperature
range as stated in vendor specs
Battery Temp
Reader
PENDING - Device Battery
Operating (21.8C)
PENDING - Device Battery
Operating (21.5C)
PENDING - Device Battery
Operating (21.6C)
General Communications 2.5.1 Validate correct frequency ranges for the chosen protocol Expected for ZigBee:
- Based on Vendor SpecsRF Sensor
PENDING
2.4 GHZ Range
PENDING
2.4 GHZ Range
PENDING
2.4 GHZ Range
General Communications 2.5.2 Validate signal strength is at optimal industry or at vendor
specified range. Note any fluctuations without obvious
interferences.
Expected for ZigBee:
- Based on Vendor Specs RF Sensor Diags state 100% Diags state 100% Diags state 100%
General Communications 2.5.3 Validate dependability of constant operation without total
loss of connectivity
Expected for ZigBee: should never totally lose connectivity if
touchscreen or hub is operating properly.
- Test Duration (1 Week) - check connection during this time.
RF SensorOngoing
(Started 10/06)
Ongoing
(Started 10/08)
Ongoing
(Started 10/08)
General Communications 2.5.4 Validate chattiness of device vs specification.
General Communications 2.5.4.a a. How often does it talk to base station? Expected for ZigBee: Frequent communication during activation and
active operation (5kb-10kb per event). Infrequent communication
when not in use (1kb-5kb/min).
Most l ikely requires a PC zigbee dongle and testing software.
Expected for Wi-Fi: Video may be constant at MB/sec versus KB/sec
per event or update.
RF SensorOngoing
(Started 10/06)
Ongoing
(Started 10/08)
Ongoing
(Started 10/08)
General Communications 2.5.5 Validate standard deviation of tests across a batch of
same devices (10+).
Determine consistency in battery drain, signal strength,
performance, etc.
Potential impact in large deployments-
Part of ongoing managed service /
Long Term Testing
Part of ongoing managed service /
Long Term Testing
Part of ongoing managed service /
Long Term Testing
General Feature/Function 2.6.1 Validate feature – Tampering Expected: A tamper message should be transmitted when device is
put into tamper mode (Based on vendor specs)-
PASSED
(When cover is removed)
PASSED
(When cover is removed)
PASSED
(When cover is removed)
General Feature/Function 2.6.2 Validate test instructions to trigger sensor.
General Feature/Function 2.6.2.a a. Were instructions clear on if and how to trigger the
sensor?
Expected: paper instructions should be included at the least.
Online instructions are a bonus. Clarity and accuracy are
mandatory.
- YESNO
(Instructions not provided)YES
General Feature/Function 2.6.2.b b. Was tripping the sensor required to activate? - YES YES YES
General Feature/Function 2.6.2.c c. If so, did the device trip on the first attempt or did it
require several attempts?
Expected: 1st Attempt (Passed)- 1ST 1ST 1ST
General Feature/Function 2.6.2.d d. Are online instructions and support available? Expected: Yes - YES NO NO
General Feature/Function 2.6.3 Validate standard deviation of tests across a batch of
same devices (10+).
Determine consistency in battery drain, signal strength,
performance, etc.
Potential impact in large deployments
-Part of ongoing managed service /
Long Term Testing
Part of ongoing managed service /
Long Term Testing
Part of ongoing managed service /
Long Term Testing
Product Certification: Device Validation (Sunny Day)
STAGE 2 : MSO Environment Validation (General) & Device Validation (Specific)
Product Certification: Device Validation - Water SensorsExecutive Summary Nov-15
All Labs - Weighted Scores Group winners highlighted in green All Labs - % of Total Group pass or fail highlights
Validation Group Comments
Device #1
Visonic Water Sensor
MCT-550 SMA
Device #2
3 Series Water
Sensor
3315-G
Device #3
SMC Water Sensor
SMCWA10-Z
Device #4
CROW Water Sensor
FLOOD-ZB
Validation Group Comments
Device #1
Visonic Water
Sensor
MCT-550 SMA
Device #2
3 Series Water
Sensor
3315-G
Device #3
SMC Water Sensor
SMCWA10-Z
Device #4
CROW Water
Sensor
FLOOD-ZB
General Packaging 53.80 49.00 55.40 53.80 General Packaging 97% 93% 99% 97%
General Physical Installation16.70 30.00 15.80 30.00
General Physical Installation62% 98% 65% 98%
General Pairing 101.00 121.00 121.00 119.00 General Pairing 92% 100% 100% 99%
General Power 80.80 81.00 81.00 73.70 General Power 90% 100% 100% 81%
General Communications 84.70 91.00 87.80 84.00 General Communications 82% 100% 88% 96%
General Device Scenarios 44.80 49.00 49.00 51.00 General Device Scenarios 84% 98% 98% 100%
General Touch Screen 71.00 71.00 71.00 71.00 General Touch Screen 100% 100% 100% 100%
General Subscriber Portal 31.00 31.00 31.00 31.00 General Subscriber Portal 100% 100% 100% 100%
General Management Portal 2.00 2.00 2.00 2.00 General Management Portal 100% 100% 100% 100%
General Mobile Application 31.00 31.00 31.00 31.00 General Mobile Application 100% 100% 100% 100%
Device Feature/Function 87.00 101.00 101.00 101.00 Device Feature/Function 93% 100% 100% 100%
Total Scores 603.80 657.00 646.00 647.50 Average 90.83% 99.02% 95.45% 97.40%
Ranking 4 1 3 2 Ranking 4 1 3 2
Identify field issues in lab before launch Eliminate vulnerabilities & attack surfaces
7
Appendix
7
pureIntegration’s IOT Labs
8
pureIntegration’s IOT Smart Lab combines formal lab booths, large test harness & demo space, office & home environments with access to the Cox QA and iControl development environments. We conduct lab testing on-site, remote on-shore, and remote off-shore as needed. Our lab environments are backed up with a datacenter with full cloud services.
Isolated Lab Booths Large Lab, Demo, Training Space Home & Office Labs Datacenter & Cloud Support
Confined testing including RF shielding Large-scale test harness tests and demos Real-world environment tests Ability to scale environments quickly
Product Certification & Testing
9
pureIntegration can accelerate new device on-boarding with the highest degree of quality while ensuring interoperability. We leverage our leading-edge MSO project knowledge, our IOT Smart Lab, MSO environments, and our extraordinary testing expertise to quickly on-board new devices for you.
1. Leverage current project knowledge from MSO home solutions, innovation center, and device deployments2. Evaluate devices in our Smart Lab with MSO devices connected to the MSO QA and Production environments3. Develop certification, baselines, entry criteria, on-boarding for current devices (sensors, door locks, screen, etc.)4. Conduct certification, baselines, entry criteria, on-boarding, endurance testing of 3rd Party WiFi devices (Nest, etc.)5. Conduct life-cycle and technical field trials including end devices and the home gateway
Value: Acceleration, Scale, Expertise, Capacity, Quality
PI Smart Lab Dev, QA & Production
MSO
IOT Project Assets IOT Testing Tools & Process
Certification Depth
10
We conduct product certification testing on MSO devices including core cases for all devices and unique cases for every device type including stress and performance tests. Our team covers the highest value tests utilizing several tools unique to IOT devices like RF, power, temperature, etc.
Product Evaluation Certification Test Cases Certification Test Results
Validation Group # Test ExpectedTool
Required
Device #1
Visonic Water Sensor
MCT-550 SMA
Device #2
3 Series Water Sensor
3315-G
Device #3
SMC Water Sensor
SMCWA10-ZGeneral Packaging 2.1.1 Validate installation instructions are easy to understand
and clearly written
Expected: paper based instructions should be included at least.
Online instructions and help are a bonus. Instructions should be
legible and should be clear in direction and context.
- Connecting Wires was a
challenge. Pairing instructions
not 100% clear and had to revisit.
FAILED PASSED
General Packaging 2.1.2 Vendor support information present. Expected: a website, phone number, email, or other method of
contact should be included.- PASSED FAILED FAILED
General Packaging 2.1.3 Validate all contents of package are present Expected: all device components should ideally be listed on the
installation instructions and of course everything should be
included in one package.
- PASSED PASSED PASSED
General Packaging 2.1.4 Validate components are securely packaged Expected: the device should have cardboard cushions or bubble
wrap if device is not durable.- PASSED PASSED PASSED
General Packaging 2.1.5 Validate battery is included Expected: batteries should be included and completely charged. - PASSED PASSED PASSED
General Packaging 2.1.5.a a. Validate battery is not actively connected to device
when shipped
Device should include a tab between the battery and contact point
to ensure no drain.- PASSED PASSED PASSED
General Packaging 2.1.5.b b. Validate battery power level is maximum when
received, if not what is the estimated power level
percentage
Expected: Battery should be between 90%-100%
-PASSED
(100%)
PASSED
(100%)
PASSED
(100%)
General Physical Installation 2.2.1 Validate ease of installation.
General Physical Installation 2.2.1.a a. How many steps are required? How many steps noted in instructions
- 5 STEPS
No instructions provided; does not
appear to require installation -
ground placement
6 STEPS
General Physical Installation 2.2.1.b b. Did it require removal of case or panel? - YES NO YES
General Physical Installation 2.2.1.c c. If yes, was the removal of case or panel simple or
difficult?
Expected: Simple or Difficult- SIMPLE N/A SIMPLE
General Pairing 2.3.1 Validate pairing compatibil ity with the system Device should share the same wireless communication protocol - ZIGBEE ZIGBEE ZIGBEE
General Pairing 2.3.2 Validate ease of connection (add a device) Expected for ZigBee: should require a single attempt once
touchscreen or hub is in activation detection mode. simply remove
strip from battery to allow it to be automatically detected.
Expected for Wi-Fi : automatically detect SSID and allow
authentication and confirmation of connectivity.
-PASSED
(SINGLE ATTEMPT)
PASSED
(SINGLE ATTEMPT)
PASSED
(SINGLE ATTEMPT)
General Pairing 2.3.2.a a. Initial operation of an activation of device Items to Capture:
- # of Steps: (standard norm is 4 steps)
- Pairing: Easy (1-3), Moderate (4-6), Difficult (7+)
- # Attempts to Pair Correctly:
-
STEPS: 6
PAIRING: MODERATE
ATTEMPTS: 1
STEPS: 6
PAIRING: EASY
ATTEMPTS: 1
STEPS: 4
PAIRING: EASY
ATTEMPTS: 1
General Pairing 2.3.2.b b. Add additional device(s), 2 or more Items to Capture:
- # of Steps:
- Pairing: Easy (1-3), Moderate (4-6), Difficult (7+)
- # Attempts to Pair Correctly:
-
STEPS: 6
PAIRING: MODERATE
ATTEMPTS: 1
STEPS: 6
PAIRING: EASY
ATTEMPTS: 2
STEPS: 4
PAIRING: EASY
ATTEMPTS: 1
General Pairing 2.3.2.c c. Did it require a complicated sequence of pressing
buttons and light sequences - YES (SEQUENCE REQ.) NO NO
General Pairing 2.3.2.d d. Did it automatically detect once battery connected? - NO (SEQUENCE REQ.) YES YES
General Pairing 2.3.2.e e. Did it require additional steps to connect to a hub or
access point?- YES (TRIPPING OF DEVICE REQ.) YES (TRIPPING OF DEVICE REQ.) YES (TRIPPING OF DEVICE REQ.)
General Pairing 2.3.2.f f. Was tripping of sensor required to complete
connection?- YES (TRIPPING OF DEVICE REQ.) YES (TRIPPING OF DEVICE REQ.) YES (TRIPPING OF DEVICE REQ.)
General Pairing 2.3.3 Validate troubleshooting. -
General Pairing 2.3.3.a a. Does it require factor re-set or reboot of the
touchscreen or hub?
Expected: simple factory re-set and re-start the activation sequence
on the touchscreen.- NO (PREMISE PASSPHRASE REQ) NO (PREMISE PASSPHRASE REQ) NO (PREMISE PASSPHRASE REQ)
General Pairing 2.3.3.b b. Did it require a manual or auto code update? - NO (PREMISE PASSPHRASE REQ) NO (PREMISE PASSPHRASE REQ) NO (PREMISE PASSPHRASE REQ)
General Pairing 2.3.3.c c. If applicable, is the factory re-set a complicated
series of pressing buttons and light sequences or was it
a one step process?
- N/A NO N/A
General Pairing 2.3.4 Validate de-pairing and re-pairing Expected for ZigBee: should require a single attempt to de-pair and
one attempt to re-pair once touchscreen or hub is in detection
mode.
-PASSED
(1 ATTEMPT)
PASSED
(1 ATTEMPT)
PASSED
(1 ATTEMPT)
General Power 2.4.1 Validate low battery Definition of low battery:
- Does the device communication or triggering fail with low battery
if not changed once notification is sent?
Good (75-99%) : device should be performed as defined on specs
Med (26-74 %), : device should be performed as defined on specs
Low ( <25? ): notification alert is sent
Battery Tester PENDING PENDING PENDING
General Power 2.4.2 Validate Dead/No Battery (<1 Day) When battery is dead for less than one day and new battery
inserted, does the device reconnect to the HUB/TS automatically?Battery Tester PENDING PENDING PENDING
General Power 2.4.3 Validate Dead/No Battery (>1 Day) When battery is dead for greater than one day and new battery
inserted, does the device reconnect to the HUB/TS automatically?Battery Tester PENDING PENDING PENDING
General Power 2.4.4 Validate performance loss of power
General Power 2.4.4.a a. Does AC power loss to TS/HUB disrupt comms
between device and TS/HUB?
Expected: No disruption of communication; battery back up will
engage.-
PASSED
(NO DISTRUPTION)
PASSED
(NO DISTRUPTION)
PASSED
(NO DISTRUPTION)
General Power 2.4.4.b b. Does AC power loss and battery power loss to
TS/HUB disrupt comms between device and TS/HUB?
Expected: Disruption of communication due to no power at TS/HUB. - PENDING PENDING PENDING
General Power 2.4.4.c c. When AC power resumes, are comms between device
and TS/HUB disrupted?
Expected: No disruption of communication, device will reconnect
automatically- PENDING PENDING PENDING
General Power 2.4.5 Validate performance loss of internet Expected: Cellular back up will kick-in and device will remain
connected- PENDING PENDING PENDING
General Power 2.4.6 Validate battery drain level – At rest / Steady State Expected: Does battery l ife last as stated in vendor specs-
Part of ongoing managed service /
Long Term Testing
Part of ongoing managed service /
Long Term Testing
Part of ongoing managed service /
Long Term Testing
General Power 2.4.7 Validate battery drain level – While communicating / non-
trigger (Check-In)
Expected: Does battery l ife last as stated in vendor specs-
Part of ongoing managed service /
Long Term Testing
Part of ongoing managed service /
Long Term Testing
Part of ongoing managed service /
Long Term Testing
General Power 2.4.8 Validate battery drain level – When triggered Expected: Does battery l ife last as stated in vendor specs-
Part of ongoing managed service /
Long Term Testing
Part of ongoing managed service /
Long Term Testing
Part of ongoing managed service /
Long Term Testing
General Power 2.4.9 Validate battery temperature Expected: Does device work as expected within battery temperature
range as stated in vendor specs
Battery Temp
Reader
PENDING - Device Battery
Operating (21.8C)
PENDING - Device Battery
Operating (21.5C)
PENDING - Device Battery
Operating (21.6C)
General Communications 2.5.1 Validate correct frequency ranges for the chosen protocol Expected for ZigBee:
- Based on Vendor SpecsRF Sensor
PENDING
2.4 GHZ Range
PENDING
2.4 GHZ Range
PENDING
2.4 GHZ Range
General Communications 2.5.2 Validate signal strength is at optimal industry or at vendor
specified range. Note any fluctuations without obvious
interferences.
Expected for ZigBee:
- Based on Vendor Specs RF Sensor Diags state 100% Diags state 100% Diags state 100%
General Communications 2.5.3 Validate dependability of constant operation without total
loss of connectivity
Expected for ZigBee: should never totally lose connectivity if
touchscreen or hub is operating properly.
- Test Duration (1 Week) - check connection during this time.
RF SensorOngoing
(Started 10/06)
Ongoing
(Started 10/08)
Ongoing
(Started 10/08)
General Communications 2.5.4 Validate chattiness of device vs specification.
General Communications 2.5.4.a a. How often does it talk to base station? Expected for ZigBee: Frequent communication during activation and
active operation (5kb-10kb per event). Infrequent communication
when not in use (1kb-5kb/min).
Most l ikely requires a PC zigbee dongle and testing software.
Expected for Wi-Fi: Video may be constant at MB/sec versus KB/sec
per event or update.
RF SensorOngoing
(Started 10/06)
Ongoing
(Started 10/08)
Ongoing
(Started 10/08)
General Communications 2.5.5 Validate standard deviation of tests across a batch of
same devices (10+).
Determine consistency in battery drain, signal strength,
performance, etc.
Potential impact in large deployments-
Part of ongoing managed service /
Long Term Testing
Part of ongoing managed service /
Long Term Testing
Part of ongoing managed service /
Long Term Testing
General Feature/Function 2.6.1 Validate feature – Tampering Expected: A tamper message should be transmitted when device is
put into tamper mode (Based on vendor specs)-
PASSED
(When cover is removed)
PASSED
(When cover is removed)
PASSED
(When cover is removed)
General Feature/Function 2.6.2 Validate test instructions to trigger sensor.
General Feature/Function 2.6.2.a a. Were instructions clear on if and how to trigger the
sensor?
Expected: paper instructions should be included at the least.
Online instructions are a bonus. Clarity and accuracy are
mandatory.
- YESNO
(Instructions not provided)YES
General Feature/Function 2.6.2.b b. Was tripping the sensor required to activate? - YES YES YES
General Feature/Function 2.6.2.c c. If so, did the device trip on the first attempt or did it
require several attempts?
Expected: 1st Attempt (Passed)- 1ST 1ST 1ST
General Feature/Function 2.6.2.d d. Are online instructions and support available? Expected: Yes - YES NO NO
General Feature/Function 2.6.3 Validate standard deviation of tests across a batch of
same devices (10+).
Determine consistency in battery drain, signal strength,
performance, etc.
Potential impact in large deployments
-Part of ongoing managed service /
Long Term Testing
Part of ongoing managed service /
Long Term Testing
Part of ongoing managed service /
Long Term Testing
Product Certification: Device Validation (Sunny Day)
STAGE 2 : MSO Environment Validation (General) & Device Validation (Specific)
Product Certification: Device Validation - Water SensorsExecutive Summary Nov-15
All Labs - Weighted Scores Group winners highlighted in green All Labs - % of Total Group pass or fail highlights
Validation Group Comments
Device #1
Visonic Water Sensor
MCT-550 SMA
Device #2
3 Series Water
Sensor
3315-G
Device #3
SMC Water Sensor
SMCWA10-Z
Device #4
CROW Water Sensor
FLOOD-ZB
Validation Group Comments
Device #1
Visonic Water
Sensor
MCT-550 SMA
Device #2
3 Series Water
Sensor
3315-G
Device #3
SMC Water Sensor
SMCWA10-Z
Device #4
CROW Water
Sensor
FLOOD-ZB
General Packaging 53.80 49.00 55.40 53.80 General Packaging 97% 93% 99% 97%
General Physical Installation16.70 30.00 15.80 30.00
General Physical Installation62% 98% 65% 98%
General Pairing 101.00 121.00 121.00 119.00 General Pairing 92% 100% 100% 99%
General Power 80.80 81.00 81.00 73.70 General Power 90% 100% 100% 81%
General Communications 84.70 91.00 87.80 84.00 General Communications 82% 100% 88% 96%
General Device Scenarios 44.80 49.00 49.00 51.00 General Device Scenarios 84% 98% 98% 100%
General Touch Screen 71.00 71.00 71.00 71.00 General Touch Screen 100% 100% 100% 100%
General Subscriber Portal 31.00 31.00 31.00 31.00 General Subscriber Portal 100% 100% 100% 100%
General Management Portal 2.00 2.00 2.00 2.00 General Management Portal 100% 100% 100% 100%
General Mobile Application 31.00 31.00 31.00 31.00 General Mobile Application 100% 100% 100% 100%
Device Feature/Function 87.00 101.00 101.00 101.00 Device Feature/Function 93% 100% 100% 100%
Total Scores 603.80 657.00 646.00 647.50 Average 90.83% 99.02% 95.45% 97.40%
Ranking 4 1 3 2 Ranking 4 1 3 2
7
Comprehensive Testing Framework
Use case & epic testing: Validate against requirements.Unit Testing: Isolated on a single app/device.
Functional Testing
Benchmarking: Create & measure against baseline metrics.Competitive analysis: Comparative data points to help validate benchmarks.
Performance Testing
Test cases: automate the keystrokes, execution, tracking, analysis, and reporting.Environment initiation: automate the spin-up of test environments and replicate as needed.
Testing Automation
Scalability: provide ability to rapidly expand testing capability without on-site constraints.Managed Environments: manage VMs and services off site to save time, resource, money.
Cloud Services
Services: ability to test integration points without having to purchase multiple licenses.Devices: ability to test full device functions via software emulation and lab tools.
Virtualization
Data center validation: Install and confirm solution in service provider data center.Market friendly validation: Install and confirm solution in friendly customer locations.
Field Testing
Interops: Host vendor test cycles for all solution components in a near-live end-to-end environment.
Interoperability Testing
Component Integration: Validate vendor components work with core solution.System Testing: End-to-End solution focused.
Integration Testing
Certification targets: Create baseline targets for common product criteria.Certification evaluation: Pre-cert and final product testing & certification.Standard conformance: Validate adherence to various platform and protocol standards.
Product Certification
Endurance & availability testing: Ensure long duration, repetitive use does not cause issues.Environmental testing: Validate in harsh conditions (temp, moisture, debris, magnetic, etc.).Break testing: Intentional disruption of the solution to ID and fix holes.
Reliability Testing
Needs assessment: Clearly define needs & requirements.Test plan & strategy: Detailed test plans and strategy.
Test Planning
Network intrusion testing: ID and plug holes in access to the IOT network.Device intrusion testing: ID and plug holes in access to devices.Data theft testing: Assess and mitigate theft of PII customer or financial data.
Security Testing
Service assurance: Confirm monitoring & dashboard reporting.Operational readiness: Confirm solution passes checklist and is included in the playbook.
Operational Testing
Release & Break/Fix: Rigorous validation that new changes do not harm existing code in production.
Regression Testing
Monitor: ability to peer, track, log, and manage devices & services.Analytics: ability to slice and process data, draw insights, make predictions and present via dashboards and executive reports.
Monitoring & Analytics
11
Security Assessment
12
pureIntegration can accelerate your security assessment focused on IOT devices. Recent DefConpresentations have publicly exposed vendor keys, creating the need to plug ZigBee, Bluetooth, Wi-Fi, and Z-Wave vulnerabilities. We will apply our assessment methodology, deep IOT device expertise, our IOT Smart Lab, and MSO labs to comprehensively assess and mitigate risks.
Value: Risk Mitigation, Expertise, Quality, Capacity, Acceleration
Establish Control Objectives
Identify the security controls your company uses (ISO, NIST, etc.)
Assist in developing an effective vulnerability management program
Implement strong access controls and security measures
Develop testing and scan schedules that fit with a patch management program
Develop an information security policy that fits your business model
Conduct readiness assessment, risk management and preparation for ISO/IEC 27001 or NIST conformance
Data Mapping & Sensitivity
Threat & Vulnerability Assessment
Control AnalysisLikelihood
Impact & Risk Analysis
Recommendation & Results
Presentation
Vulnerability & Penetration Testing Methodology
Planning Discovery & Attacks Reporting
Current Security Program
Security Strategy Program
Assess
Plan
DesignImplemen
t
Manage
Security Assessment
Report
Risk Mitigation Strategy
System Security Plan
Overview of the security requirements, agreed upon security controls, and supporting security
related documents
Security controls for logical and physical needs to protect systems and data
Measures implemented or planned to correct deficiencies and to reduce or eliminate known
vulnerabilities
Adopting a centralized security model
Developing a Comprehensive Security Strategy
Wireless Navigator & Mapping
13
pureIntegration utilizes wireless navigation tools that map out and help optimize the placement & operation of the Cox security solution. This solution has a direct impact on reducing truck rolls and trouble calls related to connectivity and performance issues. We leverage our current MSO project expertise, our IOT Smart Lab tools, and our extraordinary wireless & IOT expertise to quickly deliver solutions for you.
1. Leverage current project knowledge from MSO home solutions, innovation center, and device deployments2. Leverage multi-radio tools and methods utilized in our IOT Smart Lab to solve problems across wireless protocols3. Leverage our extraordinary depth in wireless communication and security solutions
Value: Expertise, Quality, Acceleration, Opex ReductionSpectrum Analysis Network Mapping Heat Mapping
Partner With pureIntegration
14
Reasons why our clients continue to engage us in the connected life journey:
• Dedicated & Deep Operator Expertise. We know your business like no other. pureIntegration has over a decade of systems integration and engineering expertise working for cable operators. We understand the operator business and are best qualified to tailor IOT solutions to work most effectively.
• Relevant Connected-Device & Automation Expertise. We have been developing & deploying the iControlsolution for Comcast and Cox for several years. We are iControl OpenHome partners integrating devices from other connected ecosystems to deliver automation speed to market for our clients.
• Cutting-edge IOT Expertise. We are helping to develop new IOT Framework solutions in the innovation labs and are spearheading work on new WiFi enabled and gateway devices. We know where the IOT space is headed and will remain on the leading edge to help guide our clients.
• Lab & Data Center Investments. pureIntegration has made significant investments in its data center, cloud environment, and lab in Herndon, VA in addition to local operations in Atlanta, Philly, Denver, and Sunnyvale to be responsive to its clients.
• Accountable, Value-Driven, and Innovative. We are accountable for our work and we focus on value to deliver successful, innovative results measured by satisfaction surveys in which we consistently exceed our client expectations.
Contacts
15
For questions related to pureIntegration’s IOT practice please contact your pureIntegration team below.
Chris Kocks, Account Executive & IOT Practice Lead
(678) 467-7458, [email protected]
