InterSystems presentatie: Active analytics for big data

Active Analytics for Big Data

Michael Brands

Agenda

• Big Data in Real Life

• How to approach Big Data in general

• Why is the Unstructured part of Big Data important

• Typical scenario’s for the Unstructured Part of Big Data

• Different approaches for different scenarios

• Conclusion

Big Data in real life

Ugliness is in a way superior to beauty because it lasts. Serge Gainsbourg
































































































































Ugliness is in a way superior to beauty because it lasts. Serge GainsbourgUgliness is in a way superior to beauty because it lasts. Serge Gainsbourg















































































































































Ugliness is in a way superior to beauty because it lasts. Serge














































Ugliness is in away superior to beauty because it lasts. Serge Gainsbourg


















way superior to beauty because it lasts. Serge Gainsbourg








































































What can you do with this ?

How to approach Big Data in general

Storage: scalable

transactional

secure

OLAP/Cubes: no warehousing

adaptable

Predictive Analytics:performance

selectivity

NLP/AI: domain independent

predictive

self learning

Structured

Data

Unstructured

Data

Why Analyzing unstructured Data

Study customer

behavior

interest

opinion

Analyse market

size

needs

adoption drivers

Objectify and share internal

knowledge

experts

processes


Typical Knowledge worker activity/time break down

58 % of time is spent on collecting

various tasks supported by variety of tools

0% 20% 40% 60% 80% 100%

email

information Search

Tracking Colleagues

Planning and Meeting

Producing Useful Output

Structured

Unstructured


Structured versus Unstructured Data per Activity

All knowledge oriented activities relyheavily on unstructured data


Time Spent on Structured / Unstructured Data

At least 85% of time is spent using or creating unstructured data

Typical Scenario1 : Pile of data








































































































































































































































































What to search for ?

How can you know what ‘s in

here ?

Key Words ?

Ugliness

Beauty

Gainsbourg

Ugliness(is superior to)

Beauty(because lasts)

Serge Gainsbourg

iKnowcan

create

breakthrough

applications

knowledge

unstructured

text

based on

hidden in

With iKnow you can create breakthrough applicationsby leveraging knowledge hidden in unstructured text

you

with

Typical Scenario1 : Pile of data : iKnow

Typical Scenario1: Implementation: iKnow

1 – Create a domain : logical set of sources

2 – Add Data : use the appropriate lister and loader (converter)

3 – Use the query API : queries for Entities, Dominance, Proximity Sources,

CRC’s, Summaries,

Set up metadata

Create dictionaries,

match the data

Setup Filters

Typical Scenario 2: Clear Need but no Insight

InterSystems

intersystems

Caché

Ensemble

iKnow

DeepSee

HealthShare

TrakCare

Terry Ragon

Cambridge

1978

iFind

iKnow

SQL-Querying

Free Text

Search

Typical Scenario 2: Clear Need but no Insight

Typical Scenario 2: Implementation: iFind

1 – Create an iFind index on %String or %Characterstream property

3 – Write the SQL queries of your choice

2 – Launch %BuildIndices($lb(“IndexName”)) in the case the class is

already populated

// For any persistent classClass User.Documents Extends %Persistent

{

Property text As %String(MAXLEN = "");

Index iText On (text) As %iFind.Index (INDEXOPTION = 3, LOWER = 1);;

//...

}

Select id, text from SQLUser.Documents

Where %id %FIND SQLUser.Documents_iTextfind ('drives’)

Setup

Use

Typical Scenario 2: Implementation: iFind

Take Away: Unstructured part of Big Data

Search+

Concepts

Relations

Paths

Dominance

Proximity

Summaries

Attributes

iFind

Concepts

Relations

Paths

Dominance

Proximity

Summaries

Attributes

Conclusion

In big data the biggest part of the data is

unstructured

Most tools focus on structured data and size

matters

InterSystems technology integrates superb tools

for both structured and unstructured data

analytics in a secure transactional environment

Technology

InterSystems presentatie: Active analytics for big data