Upload
embarcadero-technologies
View
592
Download
2
Embed Size (px)
Citation preview
© 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 1
Aug 26th, 2015 Webinar,
By Len Silverston, Universal Data Models, LLCSponsored by Embarcadero Technologies
© 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 2
Purpose
Share Keys
to Big Data Modeling and How to Collaborate
© 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 3
Agenda
• Big Data Overview
• Data Modeling in Big Data
• Collaboration Principles
© 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 4
Big Data
Big data is a broad term for data sets so large or
complex that traditional data
processing applications are inadequate. Wikipedia
3Vs – Volume, Velocity, VarietyBy 2020 - 44 zettabytes!
Mostly unstructured
© 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 5
Unstructured DataInformation that either does not have a pre-defined data model or is not organized in a pre-defined manner. Wikipedia
How can data have
no structure?
Is "unstructured" data
merely unmodeled?*
* Structure, Models and Meaning’ Seth Grimes, Information Week,
© 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 6
New Landscape - NoSQLKEY VALUE DATABASES
GRAPHDATABASES
DOCUMENT STORES
MongoDBMUMPS DatabaseObjectDatabase++OrientDBPostgreSQLQizxRethinkDBRocket U2SednaSimpleDB
Solr
TokuMXOpenLinkVirtuoso
OpenLinkVirtuosoOracle Spatial and GraphOracle NoSQL DatabaseOrientDBOQGRAPHProfium SenseR2DFROISSemblent Lionsgatesones GraphDBSPARQLCitySqrrl EnterpriseStardogTeradata AsterTitanTripleBitVelocityGraphVertexDBVivaceGraphWeaverWhiteDBOhmDB
RedisXAPKV - solid-state drive or rotating disk[edit]AerospikeBigTableCDBClusterpoint Database ServerCouchbase ServerFairCom c-treeACEGT.MHibariKeyspaceLevelDBLMDBMemcacheDB (using Berkeley DB or LMDB)MongoDBNoSQLzCoherenceOracle NoSQL DatabaseOpenLink VirtuosoTarantoolTokyo CabinetTuple space
KV - eventually consistentApache CassandraDynamo
Oracle NoSQL DatabaseProject VoldemortRiakOpenLink Virtuoso
KV – orderedBerkeley DBFairCom c-treeACE/c-treeRTGFoundationDBHyperDexIBM Informix C-ISAMInfinityDBLMDBMemcacheDBNDBMKV - RAM[edit]AerospikeCoherenceHazelcastmemcachedOpenLink Virtuoso
BaseX
CloudantClusterpoint DatabaseCouchbase ServerCouchDBCrateIODocumentDBElasticsearcheXist
HyperDexInformixJackrabbitLotus Notes (IBM Lotus Domino)MarkLogic
AllegroGraphArangoDB
BlazegraphBitsy
BrightstarDB
Cayley
DEX/Sparksee[2]
Filament
GraphBase
Graphd
Graph Engine[3]Grapholytic
Horton
HyperGraphDB
IBM System G Native Store
InfiniteGraph
InfoGrid
jCoreDB Graph
Neo4j
OntotextGraphDB
Orly
© 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 7
In this age of Big Data,is there ‘less of a need’ or
‘more of a need’ for data modeling?(or ‘no need’ or the ‘same need’)
© 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 8
Why Model?
8
DATA
Understand
Design?
Common semantics?
© 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 9
UNDERSTANDOVERSTAND
What are customers saying about our products?
What exactly do we mean by a customer?Is a prospect that has signed a contract but not paid yet, a customer?Is a person that only bought from us over 10 years ago a customer?Is an organization that bought a minor item from us a customer?Is sales volume based on orders, invoices, payments, or GL posts?
What are we predicting our sales volume to be this quarter?
© 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 10
REQUIRESTEXTCON
TEXT
© 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 11
Traditional
MODEL
(and DESIGN)LOAD
EXPLORE/
QUERY
DATA EXPLORE
‘Schema on write’
© 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 12
Big Data
LOAD QUERY MODEL
NoSQL STORE
EXPLORE
But Fast and Agile!
‘Schema on read’
© 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 13
Data modeling in Big Data
Customer---
NoSQL DATABASE
Documents---
Product---
Key values---
Conceptual/business data model
Understanding
Logical/physical data model
Architecture/Design
RELATIONAL DATABASE
(i.e., Data warehouse/data mart)
May transfer into structured database (using models)
© 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 14
Big Data Modeling Considerations
• Changes nature of modeling– Later– Modeling for understanding
• Design considerations - performance and scalable• Changes where physical structures reside: in code• Shifting functions to programming–Performance– Security– Integrity
• Lately, SQL interfaces over NoSQL
© 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 15
When to Model First,When To Explore First
Explore First
When format cannot be predicted in advance (Rapidly changing data structures)
When you need to keep ‘data as is’
Continually new sources of data
Don’t know if valuable (exploratory)
Huge amounts of information (e.g. streaming terabytes per minute)
E.g. Cyber terrorism, Sentiment Analysis
Model First
More predictable data structure
When there is some flexibility to modify/conform data
Stable and known sources
Know that it’s valuable
Reasonable amount of information for relational
E.g. Customer demographics, Product info, Sales History
© 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 16
What Does ‘Agile’ Mean?• Customer solution – deliver value• Flexible• Fast• Iterative• Sustainable – constant
pace• Quality design
(and efficient)
• Human Factors– Communication - face to face– Collaborative– Trust– Motivation– Ongoing reflecting and adjusting
Quotes from principles behind the Agile Manifesto can be found at http://agilemanifesto.org/principles.html
“Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.”
“Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage”
“Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale”
“Business people and developers must work together daily throughout the project.”
“Working software is the primary measure of progress.”
“Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.”
“Continuous attention to technical excellence and good design enhances agility.”
“The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.”
“Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.”
“Simplicity--the art of maximizing the amount of work not done--is essential.”
“At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.”
“"The best architectures, requirements, and designs emerge from self-organizing teams.”
Can we do this in data modeling?
© 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 17
What is Agile - What is NOT Agile?• Agile data modeling
– Deliver flexible, quality data models that facilitate sustainability and deliver value, in a quick, iterative, collaborative way, building trust and motivation
• NOT agile data modeling
– Quick & dirty
– Excuse to develop more silos
– Without quality or understanding
PERSON ORGANIZATION
PARTY
PARTY ROLE
PARTY RELATIONSHIP
PARTY CONTACT MECHANISM
FACILITY
CASE
WORK EFFORT
ROLE TYPEWORK EFFORT ROLE
CONTACT MECHANISM
SUPPLIERCUSTOMER WORKER PARTNER
CONTACT MECHANISM TYPE
PROJECTPROGRAM TASK
FIXED ASSET
ASSIGNMENT
WORK EFFORT
ASSOCATION
COMMUNICATION EVENT
FIXED ASSET
PRODUCT
GOOD SERVICE
© 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 18
How can we perform agile data modeling?
1. Re-use2. Quick broad brush data
model3. Correct model for
correct purpose4. Prioritize5. Deliver6. Understand motivations7. Have lots of choices
available
See Article: “Data Modeling’s Role in Agile Development”http://tdwi.org/articles/2010/07/07/data-modeling-agile-development.aspx
RE-USE!
© 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 19
EPISODE TYPE
EPISODE TYPE ID
DESCRIPTION
HEALTH CARE EPISODE
HEALTH CARE EPISODE ID
PATIENT PARTY ID (FK)
PATIENT ROLE TYPE ID (FK)
INCIDENT ID (FK)
EPISODE TYPE ID (FK)
EPISODE CREATE DATE
HEALTH CARE VISIT
HEALTH CARE VISIT ID
PATIENT PARTY ID (FK)
PATIENT ROLE TYPE ID (FK)
CONTACT MECHANISM ID (FK)
FACILITY ID (FK)
FROM DATE
THRU DATEINCIDENT
INCIDENT ID
INCIDENT TYPE ID (FK)
INCIDENT DATE
DESCRIPTION
EMPL RELATED IND
INCIDENT TYPEINCIDENT TYPE ID
DESCRIPTION
PATIENT
PARTY ID (FK)
ROLE TYPE ID (FK)
SYMPTOM
SYMPTOM ID
HEALTH CARE EPISODE ID (FK)
SYMPTOM TYPE ID (FK)
DESCRIPTION
SYMPTOM TYPE
SYMPTOM TYPE ID
DESCRIPTION
VISIT REASON
VISIT REASON ID
HEALTH CARE VISIT ID (FK)
SYMPTOM ID (FK)
HEALTH CARE EPISODE ID (FK)
DESCRIPTION
HEALTH CARE DELIVERY
HEALTH CARE DELIVERY ID
HEALTH CARE VISIT ID (FK)
HEALTH CARE EPISODE ID (FK)
HEALTH CARE OFFERING ID (FK)
FROM DATE
THRU DATE
DELIVERY NOTES
Re-use to understand
Universal Data Models
© 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 20
We must understand the
data and therefore
continue to develop data
models, even in this 'Big
Data' era.
Come on! Get into
the new mindset of
today’s Big Data! We
need to do things
differently today!
How can you use the data without
first understanding it?
Also, it’s important that we all
use common semantics.
Are you trying to slow
us down and continue
to try to enforce
bureaucracy?!
Data Modeler Data Scientist
© 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 21
Biggest Issue:
“Mine”
Data ‘Mine’ing
© 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 22
Key to Big Data Modeling:Data ‘Ours’ing
© 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 23
Keys To Collaboration
• Shared Purpose
• Understand Motivations
• Develop Trust
• Listen
• Manage Conflict
© 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 24
Vision?
Mission?
1. Shared Purpose
© 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 25
Can you state the exact mission statement of your organization?
(without looking it up first!)
Be Honest
BE HONEST
© 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 26
Position Versus Interest
INTERESTS A
POSITION A
INTERESTS B
POSITION B
© 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 27
2. Understand Motivations
© 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 28
Motivational Model - Sponsorship Map
© 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 29
What Does The Business Really Need?
• Insight?
• Buying behavior?
• Assessment?
• Prescriptions?
• Predictions?
© 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 30
Who is the MOST important person to know their motivations?
A. The Most Influential Sponsor?
B. Your Boss?
C. Your Most Difficult Person Who Is the Greatest Obstacle in Your Effort?
D. Yourself?
© 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 31
Core Elements of TrustKeys to Trust3. Trust
© 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 32
• Character–Integrity
–Intent
– Vulnerability/openness
• Competence–Capabilities
–ResultsFrom “The Speed Of Trust” By Stephen M. R. Covey
Keys to Trust
© 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 33
To Listen, ACCEPT
(A) ware, attention, alert
(C) are
(C) onfirm, check
(E) mpathize
(P) urpose
(T) otally (with all senses)
L
4. Listen and ACCEPT
© 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 34
Keys to Trust5. Conflict Management
© 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 35
What is the first thing to do in a conflict?
A. Define your strategy for winning?
B. Understand their perspective?
C. Don’t react?
D. Figure out a win-win?
E. Something else?
© 2014 Universal Data Models, LLC - All Rights Reserved 36
From “Getting Past No: Negotiating with Difficult People”, William Ury
Step 1.Don’t React - Observe
© 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 37
Don’t React
EventFeelings/Thoughts Emotional
PhysicalStories
Reaction
FreezeFlightFight
Mess
Step 1. Don’t react
© 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 38
Respond
EventFeelings/Thoughts
Emotional,Physical,Stories
Data
Stop – observe.Data?Questions?
Response
Intelligent Actions
Step 1. Don’t react
© 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 39
Big Data Modeling
Model To Understand – Even if after viewing data
Collaboration is the key
Find Common Purpose
Understand Motivations
Develop Trust
Listen
Conflict Management
© 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 40
What Will You Do With This?
© 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 41
Questions or More Info?
Twitter: @lensilverston
For info on template Models:www.embarcadero.com/products/er-studio-universal-data-models
© 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 42
Creative Commons Image Attributions
Much thanks to those who provided the creative common images in this presentation. Thanks for:
Stars https://www.flickr.com/photos/tom_hall_nz/17317951241/sizes/sq/ All rights reserved
by Kiwi Tom
Sky https://www.flickr.com/photos/cubagallery/9679210392 © All rights reserved
by ►CubaGallery
License