Migrating to MongoDB: Best Practices

Migrating to MongoDBBest Practices

Muthu ChinnasamySenior Solutions Architect

Agenda

• Project Team

• Schema Design

• Application Integration

• Data Migration Options

• Ops considerations

Why MongoDB

• Rich documents

• Promote business agility

• Achieve higher scalability

• Lower budget strain compared to RDBMS

RDBMS to MongoDB – Success stories

Project Team

Organizing for Success - Stakeholders

• Key to success: Involve all key stakeholders for the applicationo Line of businesso Developerso Data Architectso DBAso Systems Administratorso Security

Organizing for Success – Project Charter

• Develop project chartero Define business and technical objectiveso Define timeliness and responsibilitieso Monitor progress and address any issues

Organizing for Success – Help needed?

• Partner services and resources available from MongoDBo Community supporto Build skills and proficiency through web based trainingo Support and consulting services

Schema Design

Definitions

RDBMS MongoDB

Table Collection

Row Document

Column Field

Index Index

JOIN Embedded Document or Reference

Document Model Benefits

RDBMS MongoDB

{

_id : ObjectId("4c4ba5e5e8aabf3"),

employee_name: "Dunham, Justin",

department : "Marketing",

title : "Product Manager, Web",

report_up: "Neray, Graham",

pay_band: “C",

benefits : [

{ type : "Health",

plan : "PPO Plus" },

{ type : "Dental",

plan : "Standard" }

]

}

Schema Design – Blogging Platform

Schema Design - Indexing

• Compound Indexes• Unique indexes• Array Indexes• Text Indexes• Geospatial indexes• Sparse Indexes

Schema Design – For more details

• Afternoon session: Data modeling deep dive

• Google: "6 rules of thumb for MongoDB schema design"

• Google: "MongoDB compound index optimization"

Application Integration

Morphia

MEAN Stack

Java Python PerlRuby

MongoDB API is implemented as methods

Drivers & Ecosystem

Developer Efficiency

Object-Relational impedance?

Alter 2TB table to modify a column?

MongoDB APIs are classes and packages

Dynamic schema

Rigid schema

Modify code to use MongoDB APIs

RDBMS

MongoDB

Data Migration

Data Migration

Source DatabaseSource

Database

ApplicationApplication

Data Migration – Can you have downtime?

Source DatabaseSource

DatabaseMasterMaster ExportingExporting

ImportingImporting MasterMaster

AvailableAvailable DegradedDegraded DownDown AvailableAvailableApplication View

Time

T1

T2

T3

Data Migration – mongoimport$ mongoimport --db test --collection customers < customers.json connected to: 127.0.0.12014-11-26T08:36:47.509-0800 imported 1000 objects$ mongoMongoDB shell version: 2.6.5connecting to: test> db.customers.findOne(){

"_id" : 363862536,"first_name" : "Landon”,"last_name" : "Moore","created_date" : ISODate("2010-03-02T22:48:35Z"),"is_active" : true,"phone" : [

{ "type" : "Work”, "number" : "683-560-1311” },{ "type" : "Other”, "number" : "437-849-4219” }

],"address" : {

"street_number" : 14,"street" : "Granite","street_type" : "Way","city" : "New Jersey","zip_code" : 96881

},"company" : ”Example"

}

Data Migration – ETL tools

Source DatabaseSource Database ETLETL

Data Migration – Hadoop

Source DatabaseSource Database

jobjobjobjob

jobjobjobjob

App Driven Migration

Source DatabaseSource Database

ApplicationApplication

Data Migration - Options

Source DatabaseSource Database

SnapshotSnapshot

Continuous Sync

Batch Migration

Batch Migration

ApplicationApplication

Application Managed

Case Study

Case Study

Uses MongoDB to safeguard over 6 billion images served to millions of customers

Problem Why MongoDB Results

• 6B images, 20TB of data

• Brittle code base on top of Oracle database – hard to scale, add features

• High SW and HW costs

• JSON-based data model

• Agile, high performance, scalable

• Alignment with Shutterfly’s services-based architecture

• 80% cost reduction

• 900% performance improvement

• Faster time-to-market

• Dev. cycles in weeks vs. tens of months

• Meta data stored in XML Blobs• App responsible for content of blob

Shutterfly – Original Data store

OracleOracle

Photo ID XML Blob

1 <xml><meta-data>…</xml>

2 <xml><meta-data>…</xml>

3 <xml><meta-data>…</xml>

Schema Migration – Initial

<?xml version="1.0" encoding="utf16"?><votes> <voteItem user="00000000" vote="1" /> <voteItem user="11111111" vote="1" /> <voteItem user="22222222" vote="1" /></votes>

<?xml version="1.0" encoding="utf16"?><votes> <voteItem user="00000000" vote="1" /> <voteItem user="11111111" vote="1" /> <voteItem user="22222222" vote="1" /></votes>

Schema Migration – Phase 1

<?xml version="1.0" encoding="utf16"?><votes> <voteItem user="00000000" vote="1" /> <voteItem user="11111111" vote="1" /> <voteItem user="22222222" vote="1" /></votes>

<?xml version="1.0" encoding="utf16"?><votes> <voteItem user="00000000" vote="1" /> <voteItem user="11111111" vote="1" /> <voteItem user="22222222" vote="1" /></votes>

{ _id : "site/the3colbys/3326/_votes", "V" : 0, "cD" : "Thu Sep 23 2010 20:38:54 GMT-0700 (PDT)", "wD" : "Thu Sep 23 2010 20:38:54 GMT-0700 (PDT)", "md5" : "71199d82ee730f271feface722a74d30", "data" : "<?xml version=\"1.0\" encoding=\"utf16\"?> <votes> <voteItem user=\"00000000\" vote=\"1\" /> <voteItem user=\"11111111\" vote=\"1\" /> <voteItem user=\"22222222\" vote=\"1\" /> </votes>" }

{ _id : "site/the3colbys/3326/_votes", "V" : 0, "cD" : "Thu Sep 23 2010 20:38:54 GMT-0700 (PDT)", "wD" : "Thu Sep 23 2010 20:38:54 GMT-0700 (PDT)", "md5" : "71199d82ee730f271feface722a74d30", "data" : "<?xml version=\"1.0\" encoding=\"utf16\"?> <votes> <voteItem user=\"00000000\" vote=\"1\" /> <voteItem user=\"11111111\" vote=\"1\" /> <voteItem user=\"22222222\" vote=\"1\" /> </votes>" }

Schema Migration – Phase 2

<?xml version="1.0" encoding="utf16"?><votes> <voteItem user="00000000" vote="1" /> <voteItem user="11111111" vote="1" /> <voteItem user="22222222" vote="1" /></votes>

<?xml version="1.0" encoding="utf16"?><votes> <voteItem user="00000000" vote="1" /> <voteItem user="11111111" vote="1" /> <voteItem user="22222222" vote="1" /></votes>

{ _id : "site/the3colbys/3326/_votes", "V" : 0, "cD" : "Thu Sep 23 2010 20:38:54 GMT-0700 (PDT)", "wD" : "Thu Sep 23 2010 20:38:54 GMT-0700 (PDT)", "md5" : "71199d82ee730f271feface722a74d30", "data" : "<?xml version=\"1.0\" encoding=\"utf16\"?> <votes> <voteItem user=\"00000000\" vote=\"1\" /> <voteItem user=\"11111111\" vote=\"1\" /> <voteItem user=\"22222222\" vote=\"1\" /> </votes>" }

{ _id : "site/the3colbys/3326/_votes", "V" : 0, "cD" : "Thu Sep 23 2010 20:38:54 GMT-0700 (PDT)", "wD" : "Thu Sep 23 2010 20:38:54 GMT-0700 (PDT)", "md5" : "71199d82ee730f271feface722a74d30", "data" : "<?xml version=\"1.0\" encoding=\"utf16\"?> <votes> <voteItem user=\"00000000\" vote=\"1\" /> <voteItem user=\"11111111\" vote=\"1\" /> <voteItem user=\"22222222\" vote=\"1\" /> </votes>" }

{ _id : "site/the3colbys/3326/_votes", "V" : 0, "cD" : "Thu Sep 23 2010 20:38:54 GMT-0700 (PDT)", "wD" : "Thu Sep 23 2010 20:38:54 GMT-0700 (PDT)", "md5" : "71199d82ee730f271feface722a74d30", "votes" : { 000000000:1, 111111111:1, 222222222:1 }}

{ _id : "site/the3colbys/3326/_votes", "V" : 0, "cD" : "Thu Sep 23 2010 20:38:54 GMT-0700 (PDT)", "wD" : "Thu Sep 23 2010 20:38:54 GMT-0700 (PDT)", "md5" : "71199d82ee730f271feface722a74d30", "votes" : { 000000000:1, 111111111:1, 222222222:1 }}

Data Migration – Application driven

1. Request for photo

2. Try to read from MongoDB

3. If cache miss, read from Oracle

4. Translate document & write to MongoDB

5. Return to clientSource DatabaseSource

Database

ApplicationApplication

11

2233

44

55

Ops Considerations

Replica Sets - No downtime maintenances

Replica set provides ops agility & HA

•Database upgrades

•Hardware swaps/maintenance

•Maintenance operations

•Automatic failover

MongoDB Management Service (MMS)

Provision

Upgrade

Scale

Continuous Backup

Point-in-Time Recovery

Alerts

1. Automation 2. Backups 3. Monitoring

Cloud Managed MongoDB

36

Defense in Depth Security Architecture

Authentication•Database •LDAP •Kerberos•x.509 Certificates

Authorization•Built-in Roles•User-Defined Roles•Field-Level Redaction

Auditing•Admin operations•Queries

Encryption•Network : SSL•Disk: Partner solutions

Help available from MongoDBMongoDB Enterprise AdvancedThe best way to run MongoDB in your data center

MongoDB Management Service (MMS)The easiest way to run MongoDB in the cloud

Production SupportIn production and under control

Development SupportLet’s get you running

ConsultingWe solve problems

TrainingGet your teams up to speed.