BIGDATA - NetAppcommunity.netapp.com/fukiw75442/attachments/fukiw75442/... · 2014-09-24 · BIGDATA SCALING OUT MONGODB and Hadoop Map/Reduce (on demand) ON NETAPP CLUSTER-MODE 8.1

BIGDATASCALING OUT

MONGODB

and

Hadoop Map/Reduce (on demand)

ON

NETAPP CLUSTER-MODE 8.1

with SnapCreator

Jorge Costa, NetApp

July 2012

revision 4

Table of ContentsINTRODUCTION.............................................................................................................................................. 4

MONGODB OVERVIEW.................................................................................................................................. 4

WHO IS USING MONGODB ?......................................................................................................................... 5

THIS DOCUMENT ........................................................................................................................................... 5

SECTION: MONGODB DEPLOYMENT EXAMPLE........................................................................................6

KNOWN ISSUES OF MONGODB ON NFS.................................................................................................................................................. 6THE LIST OF HOSTS................................................................................................................................................................................... 6THE LIST OF PROCESSES AND PORTS................................................................................................................................................... 8

INSTALATION STEPS................................................................................................................................... 10

CREATE VSERVER VSERVER1................................................................................................................................................................ 10CREATE VSERVER1-NFS INTERFACE.................................................................................................................................................... 11CREATE VSERVER1-MGMT INTERFACE................................................................................................................................................. 11CREATE FLEXVOLS.................................................................................................................................................................................. 11CHANGE SNAPSHOT POLICY ................................................................................................................................................................. 11CREATE QTREES...................................................................................................................................................................................... 11ADD AGGREGATES TO VSERVER........................................................................................................................................................... 12DEFINE AN EXPORT-POLICY................................................................................................................................................................... 12INSTALL MONGODB SOFTWARE............................................................................................................................................................ 13START REPLICA SETS.............................................................................................................................................................................. 16START ARBITERS (CONFIG) SERVERS.................................................................................................................................................. 16START MONGOS ...................................................................................................................................................................................... 17BUILD REPLICA SETS.............................................................................................................................................................................. 17BUILD SHARDS......................................................................................................................................................................................... 17BUILD DATABASE LOTTERY................................................................................................................................................................... 17

SECTION: SCALING AND SHARDING.........................................................................................................18

CREATE COLLECTION TICKETS ON DATABASE LOTTERY................................................................................................................. 19CREATE INDEX ......................................................................................................................................................................................... 20ENABLE SHARDING ON COLLECTION LOTTERY.TICKETS.................................................................................................................. 20PRE-SPLIT THE SHARDS......................................................................................................................................................................... 21

SECTION: DATA LOADING EXAMPLE.........................................................................................................22

POPULATE COLLECTION LOTTERY.TICKETS....................................................................................................................................... 22INSTALL PYMONGO.................................................................................................................................................................................. 22LOAD DATA................................................................................................................................................................................................ 23CHECK UTILIZATION................................................................................................................................................................................. 23

SECTION: IMPACT OF STORAGE EFFICIENCY.........................................................................................24

ENABLE STORAGE EFFICIENCY............................................................................................................................................................. 24EXAMPLE WITH 78 WEEKS LOADED ..................................................................................................................................................... 25STORAGE EFFICIENCY SAVINGS ANALYSIS......................................................................................................................................... 26

SECTION: CONSISTENT BACKUPS ACROSS MULTIPLE SERVERS.......................................................27

INSTALL SNAPCREATOR......................................................................................................................................................................... 27CREATE SNAPCREATOR USER............................................................................................................................................................... 27CONFIGURE MONGODB BACKUP ......................................................................................................................................................... 28CONFIGURE A SNAPCREATOR PROFILE FOR BACKUP...................................................................................................................... 30TAKE A SNAPCREATOR BACKUP .......................................................................................................................................................... 31

SECTION: DATABASE CLONING FOR DEV/TEST ENVIRONMENTS........................................................35

FIX /ETC/HOSTS........................................................................................................................................................................................ 35INSTALL SNAPCREATOR AGENT ON THE CLONEDB SERVER........................................................................................................... 35PREPARE HOST ENVIRONMENT ............................................................................................................................................................ 36BUILD SNAPCREATOR PROFILES.......................................................................................................................................................... 36BUILD SNAPCREATOR CLONE FROM SNAPSHOT ............................................................................................................................. 38REVIEW CLONED DATABASE.................................................................................................................................................................. 43

SECTION:PARTIAL-RESTORE: RESTORING A SET OF DOCUMENTS....................................................44

CONNECT TO THE PRODUCTION DATABASE....................................................................................................................................... 44CHECK NUMBER OF DOCUMENTS ASSOCIATED WITH WEEK 42......................................................................................................44REMOVE ALL OBJECTS FROM THAT WEEK.......................................................................................................................................... 44COPY THE RESTORE PYTHON SCRIPT:................................................................................................................................................. 44EXECUTE THE RESTORE......................................................................................................................................................................... 45CONNECT TO PRODUCTION DATABASE............................................................................................................................................... 45CHECK FOR THE RESTORED OBJECTS................................................................................................................................................ 45

SECTION: FULL RESTORE EXAMPLE........................................................................................................46

CONNECT TO PRODUCTION, AND DROP THE TICKETS COLLECTION..............................................................................................46KILL ALL MONGODB PROCESSES......................................................................................................................................................... 46UMOUNT FILESYSTEMS ON EVERY NODE............................................................................................................................................ 46RETRIEVE THE LIST OF AVAILABLE SNAPSHOTS................................................................................................................................ 47RESTORE THE DATA FROM THE SNAPSHOTS USING SNAPCREATOR.............................................................................................47START THE MONGOD PROCESSES ACROSS ALL SERVERS.............................................................................................................. 48

SECTION: SCALING OUT ADDING NEW SHARDS ....................................................................................53

CREATE FLEXVOLS ................................................................................................................................................................................. 53CHANGE SNAPSHOT POLICY ................................................................................................................................................................. 53CREATE QTREES...................................................................................................................................................................................... 53DEFINE AN EXPORT-POLICY................................................................................................................................................................... 53INSTALL MONGODB SOFTWARE............................................................................................................................................................ 53START REPLICA MEMBERS..................................................................................................................................................................... 54BUILD REPLICA SETS.............................................................................................................................................................................. 55ADD SHARDS............................................................................................................................................................................................ 55VERIFY SHARD INFORMATION................................................................................................................................................................ 55

SECTION: RUNNING HADOOP MAP/REDUCE ON DEMAND....................................................................56

PREPARATION STEPS.............................................................................................................................................................................. 57CREATE FLEXVOLS FOR THE NAMENODE .......................................................................................................................................... 57CREATE QTREES...................................................................................................................................................................................... 58DEFINE AN EXPORT-POLICY................................................................................................................................................................... 58FIX /ETC/HOSTS ON ALL CLONED MONGODB SERVERS................................................................................................................... 58INSTALL SNAPCREATOR AGENT ON ALL CLONED MONGODB SERVERS........................................................................................58BUILD SNAPCREATOR PROFILES.......................................................................................................................................................... 59BUILD SNAPCREATOR CLONE FROM SNAPSHOT ............................................................................................................................. 71INSTALL HADDOP SOFTWARE ON HADOOP NODES AND APPSERVER............................................................................................72INSTALL HADOOP SPECIFIC PACKAGES ............................................................................................................................................ 72CONFIGURE HADOOP ............................................................................................................................................................................. 72FORMAT HDFS ......................................................................................................................................................................................... 74START THE HADOOP PROCESSES ........................................................................................................................................................ 75INSTALL MONGODB PLUGIN FOR HADOOP ......................................................................................................................................... 75RUNNING A MAP/REDUCE JOB .............................................................................................................................................................. 76

CONCLUSION................................................................................................................................................ 79

REFERENCES............................................................................................................................................... 80

INTRODUCTION

This paper provides a model of how NetApp Cluster-Mode can make easier the management of large noSQL distributed databases, such as MongoDB.

Included are methods for; reducing storage consumption, consistent backup and fast restores, cloning large number of MongoDB nodes for pre-production environments, and running large map-reduce Hadoop jobs on demand.

The content in this document should provide the reader with enough information to set up a MongoDB distributed and highly-available database on NetApp Cluster-Mode. The document provides a workflow with step-by-step tasks for a typical deployment scenario.

This document was based on information made available by 10Gen at http://www.mongodb.org/display/DOCS/Home

MONGODB OVERVIEW

MongoDB is a document-oriented noSQL distributed database, available under an GNU AGPLv 3.0 license. Development and support are (at the time of writing) only provided by 10Gen. http://www.10gen.com

MongoDB as a document-oriented database is very different from other databases such as Oracle, SQLserver or MySQL. MongoDB doesn't have the concept of a fixed schema as withrelational databases. Data in MongoDB is stored in groups of 'key:value' pairs in binary JSON objects (BSON). These are referred in MongoDB as documents; documents can be modified, extended with additional 'key:value pairs', or even include nested MongoDB documents.

MongoDB was developed with scalability and performance in mind. It can scale to a large number of servers by automatically splitting and balancing the data across all its servers.

The concept of balancing slices of the database across servers is called sharding and each server that is part of a sharded database is called a shard.

Database collections can be sharded across these shard servers based on a shard key, this key is what provides MongoDB with a method to decide where a new bit of data should be saved.

These sharded collections are then addressed as chunks (with a default size of 64MB). In an ideal world each shard would contain an identical number of chunks.

A 'balancer' process runs in the background and moves chunks between shards when these become unbalanced (in terms of numbers). It is also this process that will populate and redistribute data when new servers are added to a sharded collection.

Performance and availability are assured by using groups of master/slaves servers known as ReplicaSets. These work in a way that writes sent to the master are replicated to the slaves and reads can be performed across all servers.

By combining ReplicaSets and Shards, MongoDB allows for a highly distributed and fault-tolerant database with fast access and write operations.

Some of the challenges with MongoDB are related to the amount of disk space it requires ,which due to the use of replication its storage requirements, can be multiple times the size of the data.

Anecdotal evidence shows that that disk speed is important to users.

NetApp Cluster-Mode addresses these issues by providing storage efficiency in the form

http://www.mongodb.org/display/DOCS/Home

http://www.10gen.com/

of deduplication and compression, automatic volume sizing for disk space management, NVRAM for fast writes and FlashCache for fast reads.

WHO IS USING MONGODB ?

Craigslist, using 18servers across 2 datacenters, currently sized for 10TB.http://blog.zawodny.com/2011/05/23/speaking-at-mongosf-on-may-24th/

SAP uses MongoDB for its ECM section of their SAP's platform as a service offering.http://www.10gen.com/presentations/mongouk-2011/introduction-to-saps-java-platform-as-a-service

Forbes is currently using MongoDB for storing articles and companies' data and are working to move more of their core assets onto it.http://www.10gen.com/presentations/mongonyc-2011/forbes

MTV uses MongoDB as the repository for its next generation CMS, which will be used to distribute content for all MTV networks' websites.http://www.10gen.com/presentations/mongoboston-2011/how-mtv-leverages-mongodb-for-cms

Disney uses MongoDB as a common repository.http://www.10gen.com/presentations/mongosv-2011/a-year-with-mongodb-running-operations-to-keep-the-game-magic-alive

THIS DOCUMENT

This document is organised into sections, providing examples for different MongoDB management tasks when used with NetApp Cluster-Mode.

• MONGODB deployment example• Scaling and using Sharding• Loading data using a simulated lottery application• Benefits of NetApp Cluster-Mode Storage Efficiency• Taking consistent backups across multiple servers• Multiple server database cloning for a pre-production environment• Restore of a single document or a group of documents • Scaling out and adding new shards• Running Map/Reduce Haddop Jobs on Demand

http://www.10gen.com/presentations/mongosv-2011/a-year-with-mongodb-running-operations-to-keep-the-game-magic-alive

http://www.10gen.com/presentations/mongosv-2011/a-year-with-mongodb-running-operations-to-keep-the-game-magic-alive

http://www.10gen.com/presentations/mongoboston-2011/how-mtv-leverages-mongodb-for-cms

http://www.10gen.com/presentations/mongoboston-2011/how-mtv-leverages-mongodb-for-cms

http://www.10gen.com/presentations/mongonyc-2011/forbes

http://www.10gen.com/presentations/mongouk-2011/introduction-to-saps-java-platform-as-a-service

http://www.10gen.com/presentations/mongouk-2011/introduction-to-saps-java-platform-as-a-service

http://blog.zawodny.com/2011/05/23/speaking-at-mongosf-on-may-24th/

SECTION: MONGODB DEPLOYMENT EXAMPLE

This section contains a how-to deployment example of MongoDB served by NetApp Cluster-Mode NFS.

While this document shows deployment steps for NFS, the layout and methodology used in this example can also be used for iSCSI or FCP setups.

This documented is based on the MongoDB Sample Initial Sharding Architecture.http://www.mongodb.org/pages/viewpage.action?pageId=21268863&navigatingVersions=true

Our setup will include two replicaSets (shard1 and shard2) on which our MongoDB collection 'lottery.tickets' will be sharded.

Each of our ReplicaSets will contain three members, these will provide high-availability by automatically voting and agreeing on a primary (master) node.

The application will connect to a mongos process running locally, this mongos receives the current state and layout of the database through the config servers (arbiters). There are three config servers in our setup.

As an architectural example, the sizes mentioned in the document are simply for demonstration purposes; production deployments should reflect real data-size requirements. NetApp features such as volume autosize should be used as much as possible as these will reduce management by automatically extending the volume to prevent it getting full.

Known Issues of MongoDB on NFS

There is a bug in MongoDB with versions older than 2.2 where performance issues can occur on NFS environments.

“Don't use fallocate on NFS”https://jira.mongodb.org/browse/SERVER-4652?page=com.atlassian.jira.plugin.system.issuetabpanels%3Aall-tabpanel#issue-tabs

“When Journaling is Activated - I/O spike occurs on NFS filer and VM”https://jira.mongodb.org/browse/SERVER-4689?page=com.atlassian.streams.streams-jira-plugin%3Aactivity-stream-issue-tab#issue-tabs

SOLUTION: Use MongoDB 2.2 or higher.WORKAROUND: Disable journaling on the master nodes but keeping it on for the replicas.

The syntax used in this document is based on the MongoDB 2.0.2 release. Please check the MongoDB documentation for changes when using 2.2 or higher releases.

The list of hosts

Below is the list of hosts used in this document. Eight nodes are used in total for the MongoDB install. As previously mentioned, the database will have two shards (shard1, shard2), each containing a ReplicaSet of three members each.

Shard1

https://jira.mongodb.org/browse/SERVER-4689?page=com.atlassian.streams.streams-jira-plugin%3Aactivity-stream-issue-tab#issue-tabs

https://jira.mongodb.org/browse/SERVER-4689?page=com.atlassian.streams.streams-jira-plugin%3Aactivity-stream-issue-tab#issue-tabs

https://jira.mongodb.org/browse/SERVER-4652?page=com.atlassian.jira.plugin.system.issuetabpanels%3Aall-tabpanel#issue-tabs

https://jira.mongodb.org/browse/SERVER-4652?page=com.atlassian.jira.plugin.system.issuetabpanels%3Aall-tabpanel#issue-tabs

http://www.mongodb.org/pages/viewpage.action?pageId=21268863&navigatingVersions=true


Contains a replicaset with the same name (shard1), this replicatSet contains members:• mongodb-shard1a• mongodb-shard1b• mongodb-shard1c

Shard2Contains a replicaset with the same name (shard2), this replicatSet contains members:

• mongodb-shard2a• mongodb-shard2b• mongodb-shard2c

We have 3 MongoDB config servers (or arbiters) which are aliases to existing nodes.• Mongodb-config1 (mongodb-shard1a)• mongodb-config2 (mongodb-shard2a)• mongodb-config3 (mongodb-shard3a)

Server appserver is the application layer, or in other words, the server from where we will be loading data.

Server mongodb-clonedb is where we will perform cloning operations, such as building a pre-production environment

SnapCreator Framework (www.snapcreator.com) will be used to take application consistent backups and perform cloning to pre-production environmentsNetApp Snap Creator Framework is OS-independent software that integrates with NetApp data protection technologies to provide storage management for your customised applications. It provides plug-ins to integrate NetApp features with popular third-party applications such as Oracle®, MySQL, and DB2.

Snap Creator Framework allows the development of customer plug-ins to provide backup procedures for applications not covered by existing supported applications. It is available free-of-charge on the NOW website, and fully supported by NetApp Global Services.

In this example. all data will be accessed through NFS on NetApp Cluster-Mode vServer vserver1-nfs interface. Management for this vServer will be done through cmode81a-cluster-mgmt.

SnapCreator commands are sent through vServer vserver1-mgmt interface.

#ipaddress #hostname #alias172.16.1.88 Appserver

172.16.1.91 mongodb-shard1a mongodb-config1

172.16.1.92 mongodb-shard1b

172.16.1.93 mongodb-shard1c

172.16.1.94 mongodb-shard2a mongodb-config2

172.16.1.95 mongodb-shard2b

172.16.1.96 mongodb-shard2c mongodb-config3

172.16.1.60 Cmode81a-cluster-mgmt

172.16.1.62 vserver1-nfs

172.16.1.64 vserver1-mgmt

172.16.1.101 mongodb-clonedb

http://www.snapcreator.com/

The list of processes and ports

In order to facilitate cloning of all of the hosts, each mongod shard process will be running with a different port number. This will allow us to start all different shard processes in the same clone host.

Task Process Running on: Alias to: Port numberShard1 mongod mongodb-shard1a 30011

Shard1 mongod mongodb-shard1b 30012

Shard1 mongod mongodb-shard1c 30013

Shard2 mongod mongodb-shard2a 30021

Shard2 mongod mongodb-shard2b 30022

Shard2 mongod mongodb-shard2c 30023

Arbiter mongod mongodb-config1 mongodb-shard1a 27019

Arbiter mongod mongodb-config2 mongodb-shard2a 27019

Arbiter mongod mongodb-config3 mongodb-shard2c 27019

mongos mongos appserver 27017

Snapcreator Server ScServer Appserver 8080

SnapCreator Agent ScAgent mongodb-clonedb 9090

Our MongoDB landscape

INSTALATION STEPS

This section documents the tasks required to deploy MongoDB,

Create vserver vserver1

NetApp Cluster-Mode provides a fully scalable storage architecture providing the necessary disk storage backend for MongoDB. The steps that follow describe the allocation of NFS shares across all MongoDB servers.

cmode81::> vserver setup Welcome to the Vserver Setup Wizard, which will lead you through the steps to create a virtual storage server that serves data to clients.

You can enter the following commands at any time: "help" or "?" if you want to have a question clarified, "back" if you want to change your answers to previous questions, and "exit" if you want to quit the Vserver Setup Wizard. Any changes you made before typing "exit" will be applied.

You can restart the Vserver Setup Wizard by typing "vserver setup". To accept a default or omit a question, do not enter a value.

Step 1. Create a Vserver. You can type "back", "exit", or "help" at any question.

Enter the Vserver name: vserver1Choose the Vserver data protocols to be configured {nfs, cifs, fcp, iscsi}: nfs, iscsi Choose the Vserver client services to be configured {ldap, nis, dns}: dns Enter the Vserver's root volume aggregate [aggr1]: Enter the Vserver language setting, or "help" to see all languages [C]: Vserver creation might take some time to finish....

Vserver vserver1 with language set to C created. The permitted protocols are nfs,iscsi.

Step 2: Create a data volume You can type "back", "exit", or "help" at any question.

Do you want to create a data volume? {yes, no} [yes]: no

Step 3: Create a logical interface. You can type "back", "exit", or "help" at any question.

Do you want to create a logical interface? {yes, no} [yes]: no

Step 4: Configure DNS (Domain Name Service). You can type "back", "exit", or "help" at any question.

Do you want to configure DNS? {yes, no} [yes]: Enter the comma separated DNS domain names: vmwarelab Enter the comma separated DNS server IP addresses: 172.16.1.1

DNS for Vserver vserver1 is configured.

Step 5: Configure NFS. You can type "back", "exit", or "help" at any question.

NFS configuration for Vserver vserver1 created successfully.

Step 6: Configure iSCSI. You can type "back", "exit", or "help" at any question. Do you want to configure iSCSI? {yes, no} [yes]: no

Vserver vserver1, with protocol(s) nfs, and service(s) dns has been configured successfully.

Create vserver1-nfs interface

All the different servers will mount NFS shares through this interface

cmode81::> network interface create -vserver vserver1 -lif vserver1-nfs -role data -data-protocol nfs -home-node cmode81-01 -home-port e0c -address 172.16.1.62 -netmask 255.255.255.0 -routing-group d172.16.1.0/24 -status-admin up -failover-policy nextavail -firewall-policy mgmt -auto-revert false

Create vserver1-mgmt interface

SnapCreator Server will use this interface for storage operations.

cmode81::> network interface create -vserver vserver1 -lif vserver1-mgmt -role data -data-protocol none -home-node cmode81-01 -home-port e0c -address 172.16.1.64 -netmask 255.255.255.0 -routing-group d172.16.1.0/24 -status-admin up -failover-policy nextavail -firewall-policy mgmt -auto-revert false

Create flexvols

Each MongoDB ReplicaSet will be allocated one FlexVol. In this setup there will be three servers writing to the same flexvol. Each server will have a dedicated qtree.

The advantage of this design is that it allows for storage efficiency to provide larger storage savings; disk space utilisation using dedup and compression can be reduced to a third.

cmode81::> volume create -vserver vserver1 -volume mongodb_shard1 -aggregate aggr1 -size 3G -state online -type RW -policy mongodb_shard1 -user root -security-style unix -autosize true -space-guarantee none -unix-permissions 0777 -junction-path /vol/mongodb_shard1[Job 37] Job succeeded: Successful

cmode81::> volume create -vserver vserver1 -volume mongodb_shard2 -aggregate aggr1 -size 3G -state online -type RW -policy mongodb_shard1 -user root -security-style unix -autosize true -space-guarantee none -unix-permissions 0777 -junction-path /vol/mongodb_shard2[Job 37] Job succeeded: Successful

Change snapshot policy

Scheduled snapshots should be disabled as we'll be taking MongoDB consistent snapshots through SnapCreator.

cmode81::> vol modify -volume mongodb_shard1 -snapshot-policy nonecmode81::> vol modify -volume mongodb_shard2 -snapshot-policy none

Create qtrees

Each replicaSet member will mount its /data/db directories from one dedicated qtree. Each flexvol contains 3 qtrees (one for each ReplicaSet Member).cmode81::> qtree create -volume mongodb_shard1 -qtree mongodb_shard1a -security-style unix -oplock-mode enable -vserver vserver1

cmode81::> qtree create -volume mongodb_shard1 -qtree mongodb_shard1b -security-style unix -oplock-mode enable -vserver vserver1 cmode81::> qtree create -volume mongodb_shard1 -qtree mongodb_shard1c -security-style unix -oplock-mode enable -vserver vserver1

cmode81::> qtree create -volume mongodb_shard2 -qtree mongodb_shard2a -security-style unix -oplock-mode enable -vserver vserver1

cmode81::> qtree create -volume mongodb_shard2 -qtree mongodb_shard2b -security-style unix -oplock-mode enable -vserver vserver1

cmode81::> qtree create -volume mongodb_shard2 -qtree mongodb_shard2c -security-style unix -oplock-

mode enable -vserver vserver1

Add Aggregates to vserver

Add the aggregate to the vserver1, this is required for snapcreator clone operations.

cmode81::> vserver show -vserver vserver1 -fields aggr-list vserver aggr-list -------- --------- vserver1 -

cmode81::> vserver modify -vserver vserver1 -aggr-list aggr1

cmode81::> vserver show -vserver vserver1 -fields aggr-list vserver aggr-list -------- --------- vserver1 aggr1

cmode81::> volume show Vserver Volume Aggregate State Type Size Available Used% --------- ------------ ------------ ---------- ---- ---------- ---------- ----- cmode81-01 vol0 aggr0 online RW 851.5MB 338.1MB 60% vserver1 cl_master_mongodb_shard1_20120221075326 aggr1 online RW 3GB 2.35GB 21% vserver1 mongodb_shard1 aggr1 online RW 3GB 2.35GB 21% vserver1 mongodb_shard2 aggr1 online RW 3GB 2.53GB 15%

Define an export-policy

The NFS export policies are set so that each Shard members only have access to its own flexvol.

cmode81::> vserver export-policy create -policyname mongodb_shard1 -vserver vserver1

cmode81::> vserver export-policy rule create -policyname mongodb_shard1 -vserver vserver1 -ruleindex 1 -protocol nfs3 -clientmatch mongodb-shard1a -rorule any -rwrule any -anon 0 -superuser any

cmode81::> vserver export-policy rule create -policyname mongodb_shard1 -vserver vserver1 -ruleindex 2 -protocol nfs3 -clientmatch mongodb-shard1b -rorule any -rwrule any -anon 0 -superuser any

cmode81::> vserver export-policy rule create -policyname mongodb_shard1 -vserver vserver1 -ruleindex 3 -protocol nfs3 -clientmatch mongodb-shard1c -rorule any -rwrule any -anon 0 -superuser any

cmode81::> vserver export-policy create -policyname mongodb_shard2 -vserver vserver1




cmode81::vserver> vserver export-policy rule show -vserver vserver1 -policyname default -fields vserver,policyname,ruleindex,protocol,clientmatch,rorule,rwrule,anon,superuser,allow-suid,allow-dev vserver policyname ruleindex protocol clientmatch rorule rwrule anon superuser allow-suid allow-dev -------- ---------- --------- -------- ----------- ------ ------ ---- --------- ---------- --------- vserver1 default 1 nfs3 0.0.0.0/0 any never 0 any true true

cmode81::vserver> vserver export-policy rule show -vserver vserver1 -policyname mongodb_shard1 -fields vserver,policyname,ruleindex,protocol,clientmatch,rorule,rwrule,anon,superuser,allow-suid,allow-dev vserver policyname ruleindex protocol clientmatch rorule rwrule anon superuser allow-suid allow-dev -------- -------------- --------- -------- ----------- ------ ------ ----- --------- ---------- --------- vserver1 mongodb_shard1 2 nfs3 mongodb-shard1a any any 0 any true true

vserver1 mongodb_shard1 3 nfs3 mongodb-shard1b any any 0 any true true vserver1 mongodb_shard1 4 nfs3 mongodb-shard1c any any 0 any true true 3 entries were displayed.

cmode81::vserver> vserver export-policy rule show -vserver vserver1 -policyname mongodb_shard2 -fields vserver,policyname,ruleindex,protocol,clientmatch,rorule,rwrule,anon,superuser,allow-suid,allow-dev vserver policyname ruleindex protocol clientmatch rorule rwrule anon superuser allow-suid allow-dev -------- -------------- --------- -------- ----------- ------ ------ ---- --------- ---------- --------- vserver1 mongodb_shard2 1 nfs3 mongodb-shard2a any any 0 any true true vserver1 mongodb_shard2 2 nfs3 mongodb-shard2b any any 0 any true true vserver1 mongodb_shard2 3 nfs3 mongodb-shard2c any any 0 any true true 3 entries were displayed.

The list of policies applied to each FlexVol

server flexvol qtreeol policy rule

mongodb-shard1a /vol/mongodb-shard1 mongodb-shard1a mongodb_shard1 1

mongodb-shard1b /vol/mongodb-shard1 mongodb-shard1b mongodb_shard1 2

mongodb-shard1c /vol/mongodb-shard2 mongodb-shard1c mongodb_shard1 3

mongodb-shard2a /vol/mongodb-shard2 mongodb-shard2a mongodb_shard2 1

mongodb-shard2b /vol/mongodb-shard2 mongodb-shard2b mongodb_shard2 2

mongodb-shard2c /vol/mongodb-shard2 mongodb-shard2c mongodb_shard2 3

Install MongoDB software

With storage allocated for the databases, we can now proceed with the software instalation across all servers.

MONGODB-SHARD1A

Create mongodb user

[root@mongodb-shard1a /]# groupadd -g 8000 mongodb [root@mongodb-shard1a /]# useradd -u 8000 -g 8000 -d /data mongodb

Edit /etc/fstab and include the mountpoints for the mongodb datafilesvserver1-nfs:/vol/mongodb_shard1/mongodb_shard1a /data nfs rw,vers=3,rsize=65535,wsize=65535,hard,proto=tcp,timeo=600,nolock,retrans=2,sec=sys,addr=172.16.1.91 0 0

[root@mongodb-shard1a /]# mount /data [root@mongodb-shard1a /]# mkdir -p /data/db/ [root@mongodb-shard1a /]# chmod -R 700 /data [root@mongodb-shard1a /]# chown -R mongodb:mongodb /data

Install mongodb software

[root@mongodb-shard1a /]# cd /opt [root@mongodb-shard1a /]# wget http://fastdl.mongodb.org/linux/mongodb-linux-x86_64-2.0.2.tgz [root@mongodb-shard1a /]# tar xzf mongodb-linux-x86_64-2.0.2.tgz [root@mongodb-shard1a /]# ln -s mongodb-linux-x86_64-2.0.2 mongodb

MONGODB-SHARD1B

Create mongodb user

[root@mongodb-shard1b /]# groupadd -g 8000 mongodb [root@mongodb-shard1b /]# useradd -u 8000 -g 8000 -d /data mongodb

edit /etc/fstab and include the mountpoints for the mongodb datafiles

vserver1-nfs:/vol/mongodb_shard1/mongodb_shard1b /data nfs rw,vers=3,rsize=65535,wsize=65535,hard,proto=tcp,timeo=600,nolock,retrans=2,sec=sys,addr=172.16.1.91 0 0

[root@mongodb-shard1b /]# mount /data [root@mongodb-shard1b /]# mkdir -p /data/db/ [root@mongodb-shard1b /]# chmod -R 700 /data [root@mongodb-shard1b /]# chown -R mongodb:mongodb /data


[root@mongodb-shard1b /]# cd /opt [root@mongodb-shard1b /]# wget http://fastdl.mongodb.org/linux/mongodb-linux-x86_64-2.0.2.tgz [root@mongodb-shard1b /]# tar xzf mongodb-linux-x86_64-2.0.2.tgz [root@mongodb-shard1b /]# ln -s mongodb-linux-x86_64-2.0.2 mongodb

MONGODB-SHARD1C

Create MongoDB user

[root@mongodb-shard1c /]# groupadd -g 8000 mongodb [root@mongodb-shard1c /]# useradd -u 8000 -g 8000 -d /data mongodb


vserver1-nfs:/vol/mongodb_shard1/mongodb_shard1c /data nfs rw,vers=3,rsize=65535,wsize=65535,hard,proto=tcp,timeo=600,nolock,retrans=2,sec=sys,addr=172.16.1.91 0 0

[root@mongodb-shard1c /]# mount /data [root@mongodb-shard1c /]# mkdir -p /data/db/ [root@mongodb-shard1c /]# chmod -R 700 /data [root@mongodb-shard1c /]# chown -R mongodb:mongodb /data

Install mongodb software[root@mongodb-shard1c /]# cd /opt [root@mongodb-shard1c /]# wget http://fastdl.mongodb.org/linux/mongodb-linux-x86_64-2.0.2.tgz [root@mongodb-shard1c /]# tar xzf mongodb-linux-x86_64-2.0.2.tgz [root@mongodb-shard1c /]# ln -s mongodb-linux-x86_64-2.0.2 mongodb

MONGODB-SHARD2A

Create mongodb user[root@mongodb-shard2a /]# groupadd -g 8000 mongodb [root@mongodb-shard2a /]# useradd -u 8000 -g 8000 -d /data mongodb


vserver1-nfs:/vol/mongodb_shard1/mongodb_shard2a /data nfs rw,vers=3,rsize=65535,wsize=65535,hard,proto=tcp,timeo=600,nolock,retrans=2,sec=sys,addr=172.16.1.91 0 0

[root@mongodb-shard2a /]# mount /data [root@mongodb-shard2a /]# mkdir -p /data/db/ [root@mongodb-shard2a /]# chmod 700 -R /data [root@mongodb-shard2a /]# chown mongodb:mongodb -R /data


[root@mongodb-shard2a /]# cd /opt [root@mongodb-shard2a /]# wget http://fastdl.mongodb.org/linux/mongodb-linux-x86_64-2.0.2.tgz [root@mongodb-shard2a /]# tar xzf mongodb-linux-x86_64-2.0.2.tgz [root@mongodb-shard2a /]# ln -s mongodb-linux-x86_64-2.0.2 mongodb

MONGODB-SHARD2B

Create mongodb user


edit /etc/fstab and include the mountpoints for the mongodb datafilesvserver1-nfs:/vol/mongodb_shard1/mongodb_shard2b /data nfs rw,vers=3,rsize=65535,wsize=65535,hard,proto=tcp,timeo=600,nolock,retrans=2,sec=sys,addr=172.16.1.91 0 0



[root@mongodb-shard2b /]# cd /opt [root@mongodb-shard2b /]# rm -rf /opt/mongo* [root@mongodb-shard2b /]# wget http://fastdl.mongodb.org/linux/mongodb-linux-x86_64-2.0.2.tgz [root@mongodb-shard2b /]# tar xzf mongodb-linux-x86_64-2.0.2.tgz [root@mongodb-shard2b /]# ln -s mongodb-linux-x86_64-2.0.2 mongodb

MONGODB-SHARD2C

Create mongodb user

[root@mongodb-shard2c /]# groupadd -g 8000 mongodb [root@mongodb-shard2c /]# useradd -u 8000 -g 8000 -d /data mongodb

Edit /etc/fstab and include the mountpoints for the mongodb datafiles

vserver1-nfs:/vol/mongodb_shard1/mongodb_shard2c /data nfs rw,vers=3,rsize=65535,wsize=65535,hard,proto=tcp,timeo=600,nolock,retrans=2,sec=sys,addr=172.16.1.91 0 0



[root@mongodb-shard2c /]# cd /opt [root@mongodb-shard2c /]# wget http://fastdl.mongodb.org/linux/mongodb-linux-x86_64-2.0.2.tgz [root@mongodb-shard2c /]# tar xzf mongodb-linux-x86_64-2.0.2.tgz [root@mongodb-shard2c /]# ln -s mongodb-linux-x86_64-2.0.2 mongodb

APPSERVER

Create mongodb user

[root@mongodb-appserver /]# groupadd -g 8000 mongodb [root@mongodb-appserver /]# useradd -u 8000 -g 8000 -d /data mongodb [root@mongodb-appserver /]# chown mongodb:mongodb /data


[root@mongodb-appserver /]# cd /opt[root@mongodb-appserver /]# wget http://fastdl.mongodb.org/linux/mongodb-linux-x86_64-2.0.2.tgz [root@mongodb-appserver /]# tar xzf mongodb-linux-x86_64-2.0.2.tgz [root@mongodb-appserver /]# ln -s mongodb-linux-x86_64-2.0.2 mongodb

Start replica sets

MONGODB-SHARD1A [root@mongodb-shard1a /]# su - mongodb [mongodb@mongodb-shard1a /]> /opt/mongodb/bin/mongod --replSet shard1/mongodb-shard1b:30012,mongodb-shard1c:30013 --journal --rest --shardsvr –-port 30011 --fork --logpath /data/db/mongod.log

MONGODB-SHARD1B [root@mongodb-shard1b /]# su - mongo [mongodb@mongodb-shard1b /]> /opt/mongodb/bin/mongod --replSet shard1/mongodb-shard1a:30011,mongodb-shard1c:30012 --journal --rest --shardsvr –-port 30012 --fork --logpath /data/db/mongod.log

MONGODB-SHARD1C [root@mongodb-shard1c /]# su - mongo[mongodb@mongodb-shard1c /]> /opt/mongodb/bin/mongod --replSet shard1/mongodb-shard1b:30012,mongodb-shard1a:30011 --journal --rest --shardsvr –-port 30013 --fork --logpath /data/db/mongod.log

MONGODB-SHARD2A

[root@mongodb-shard2a /]# su - mongodb [mongodb@mongodb-shard2a /]> /opt/mongodb/bin/mongod --replSet shard2/mongodb-shard2b:30022,mongodb-shard2c:30023 --journal --rest --shardsvr –-port 30021 --fork --logpath /data/db/mongod.log

MONGODB-SHARD2B

[root@mongodb-shard2b /]# su - mongodb [mongodb@mongodb-shard2b /]> /opt/mongodb/bin/mongod --replSet shard2/mongodb-shard2a:30021,mongodb-shard2c:30023 --journal --rest --shardsvr –-port 30022 --fork --logpath /data/db/mongod.log

MONGODB-SHARD2C

[root@mongodb-shard2c /]# su - mongodb [mongodb@mongodb-shard2c /]> /opt/mongodb/bin/mongod --replSet shard2/mongodb-shard1b:30022,mongodb-shard2a:30021 --journal --rest --shardsvr –-port 30023 --fork --logpath /data/db/mongod.log

Start arbiters (config) servers

MONGODB-SHARD1A

[root@mongodb-shard1a /]# su - mongodb [mongodb@mongodb-shard1a /]> mkdir /data/configdb [mongodb@mongodb-shard1a /]> /opt/mongodb/bin/mongod --configsvr --port 27019 --fork --logpath /data/db/arbiter.log

MONGODB-SHARD2A

[root@mongodb-shard2a /]# su - mongodb [mongodb@mongodb-shard2a /]> mkdir /data/configdb [mongodb@mongodb-shard2a /]> /opt/mongodb/bin/mongod --configsvr --port 27019 --fork --logpath /data/db/arbiter.log

MONGODB-SHARD2C

[root@mongodb-shard1c /]# su - mongodb [mongodb@mongodb-shard1c /]> mkdir /data/configdb [mongodb@mongodb-shard1c /]> /opt/mongodb/bin/mongod --configsvr --port 27019 --fork --logpath /data/db/arbiter.log

Start mongos

APPSERVER

[root@appserver /]# su - mongodb [mongodb@appserver /]> /opt/mongodb/bin/mongos --configdb mongodb-config1:27019,mongodb-config2:27019,mongodb-config3:27019 –logpath /data/mongos.log –fork –port 27017

Build replica sets

Connect to application server APPSERVER

[mongodb@appserver /]> /opt/mongodb/bin/mongo mongodb-shard1a:30011/admin cfg = { _id : "shard1", members : [ {_id : 0, host : "mongodb-shard1a:30011"}, {_id : 1, host : "mongodb-shard1b:30012"}, {_id : 2, host : "mongodb-shard1c:30013"} ] }

rs.initiate(cfg); exit;

[mongodb@appserver /]> /opt/mongodb/bin/mongo mongodb-shard2a:30021/admin cfg = { _id : "shard2", members : [ {_id : 0, host : "mongodb-shard2a:30021"}, {_id : 1, host : "mongodb-shard2b:30022"}, {_id : 2, host : "mongodb-shard2c:30023"} ] }

rs.initiate(cfg);exit;

Build shards

Connect to APPSERVER[root@appserver /]> su - mongodb [mongodb@appserver /]> /opt/mongodb/bin/mongo appserver:27017/admin

db.adminCommand( { addShard : "shard1/mongodb-shard1a:30011,mongodb-shard1b:30012,mongodb-shard1c:30013" } ) db.adminCommand( { addShard : "shard2/mongodb-shard2a:30021,mongodb-shard2b:30022,mongodb-shard2c:30023" } )

Build Database lottery

To simulate real data, we'll be creating a database collection to store lottery tickets entries.

We'll be running a python script which will generate random lottery choices (8 'X's) and populate the database with those lottery tickets.

The document layout for our collection 'lottery.tickets' looks like the following:week:integer agency:integer ticketnumber:integer luckynumbers:{a,b,c,d,e,f,g,h} lnhash:integer (luckynumbers multiplied)

We will be keeping track of the week, the agency which submitted the lottery ticket, the 'ticketnumber', the 'luckynumbers' contained in that lottery ticket, and will index and shard this collection based on a key which is the product of all 'luckynumbers' combined, together with the week number.

SECTION: SCALING AND SHARDING

Sharding works by splitting a data range into several chunks all across its shards, ideally a shard key, should reflect the queries performed against that collection, and provide a finite source of data values.

Incremental keys, such as week number, are not recommended for shard-keys as they'll force new writes into the last added shard, creating storage complications, inefficient balancing and IO performance issues.

By choosing lnhash combined with the week number for our shard key, we provide MongoDB with a method for distributing chunks across all shards while still maintaining locality, which will improve good index reads and query times.

With the example provided, imagine that our lottery tickets contain between 1 and 48 holes, and that any ticket contains eight marked holes. The values of the shard key will range from:

[1][2][3][4][5][6][7][8] : 40320, week 1

to

[41][42][43][44][45][46][47][48]: 15214711438080, week 48

By indexing on lnhash, we can quickly find out who the lucky winner of this week is by simply multiplying this week's lucky numbers, and performing a search against that product.

Let's imagine that the lucky numbers this week are: 23,17,4,33,1,5,23,16

Which translates to a lnhash of 94966080use lotterydb.tickets.find({lnhash:'94966080'}, {week:'90'})

This will provide very fast query results as the current weeks data is likely to be still sitting

in RAM (our current working set), and due to the way sharding works, the mongos process will direct the query to the shard server(s) containing that range of data.

In our case there will be high locality of data as we store all identical lottery tickets keys together in the same chunk.

If we were looking to gather all the lucky-number entries for the last week, then we could execute:use lotterydb.tickets.find({week:'90'})

This query would look up the index (we're indexing on week), and query every single shard as the chunks for any single week should be equally distributed across all servers.

Connect to APPSERVER [root@appserver /]> su - mongodb [mongodb@appserver /]> /opt/mongodb/bin/mongo appserver:27017/admin

Create collection Tickets on database lotteryMongoDB shell version: 2.0.2 connecting to: appserver:27017/test mongos> use lottery switched to db lottery mongos>

mongos> db.createCollection("tickets") { "ok" : 1 } mongos>

#lnhash -> lucky numbers hash db.tickets.insert( { lnhash: 40320, week : 1, agency: 1, ticketnumber: 1, luckynumbers : [ 1,2,3,4,5,6,7,8 ] })

mongos> db.tickets.findOne() { "_id" : ObjectId("4f2b2850193f9ed1debd72df"), "lnhash" : 40320, "week" : 1, "agency" : 1, "ticketnumber" : 1, "luckynumbers" : [ 1, 2, 3, 4, 5, 6, 7, 8 ] }

Create index

mongos> use lottery switched to db lottery mongos> db.tickets.ensureIndex({lnhash:1, week:1}); mongos>

mongos> db.tickets.getIndexes() [ { "v" : 1, "key" : { "_id" : 1 }, "ns" : "lottery.tickets", "name" : "_id_" }, { "v" : 1, "key" : { "lnhash" : 1, "week" : 1 }, "ns" : "lottery.tickets", "name" : "lnhash_1_week_1" } ] mongos>

ENABLE SHARDING on collection lottery.ticketsmongos> use admin switched to db admin mongos> db.runCommand( { enablesharding : "lottery" } ); { "ok" : 1 } mongos>

mongos> db.runCommand( { enablesharding : "tickets" } ); { "ok" : 1 }

mongos> use admin switched to db admin mongos> mongos> db.runCommand( { shardcollection : "lottery.tickets", key : { lnhash:1, week:1 } }); { "collectionsharded" : "lottery.tickets", "ok" : 1 } mongos>

mongos> db.printShardingStatus() --- Sharding Status --- sharding version: { "_id" : 1, "version" : 3 } shards: { "_id" : "shard1", "host" : "shard1/mongodb-shard1a:30011,mongodb-shard1a:27018,mongodb-shard1c:30013,mongodb-shard1b:30012" } { "_id" : "shard2", "host" : "shard2/mongodb-shard2a:30021,mongodb-shard2c:30023,mongodb-shard2b:30022" } databases: { "_id" : "admin", "partitioned" : false, "primary" : "config" } { "_id" : "test", "partitioned" : false, "primary" : "shard1" } { "_id" : "tickets", "partitioned" : true, "primary" : "shard2" } { "_id" : "lottery", "partitioned" : true, "primary" : "shard1" } lottery.tickets chunks: shard1 1 { "lnhash" : { $minKey : 1 }, "week" : { $minKey : 1 } } -->> { "lnhash" : { $maxKey : 1 }, "week" : { $maxKey : 1 } } on : shard1 { "t" : 1000, "i" : 0 }

mongos>

Pre-split the shards

Build two chunks to start, one on each shardmongos> use admin mongos> db.runCommand({ split : "lottery.tickets", middle : { lnhash : "7607355719040" } })mongos> db.runCommand({ moveChunk: "lottery.tickets", middle : { lnhash : "7607355719041" }, to: “shard2” })

SECTION: DATA LOADING EXAMPLE

Populate collection lottery.tickets

Check used space

First, lets note down of how much disk space we are using with an empty database.cmode81::> df -m Filesystem total used avail capacity Mounted on /vol/mongodb_shard1/ 2918MB 589MB 2329MB 20% /vol/mongodb_shard1 /vol/mongodb_shard1/.snapshot 153MB 0MB 153MB 0% /vol/mongodb_shard1/.snapshot/vol/mongodb_shard2/ 2918MB 589MB 2329MB 20% /vol/mongodb_shard2 /vol/mongodb_shard2/.snapshot 153MB 0MB 153MB 0% /vol/mongodb_shard2/.snapshot

INSTALL PYMONGO

On APPSERVER,

Install the MONGODB driver for python[root@appserver /]> yum install python-setuptools.noarch

[root@appserver /]# easy_install pymongo Searching for pymongo Reading http://pypi.python.org/simple/pymongo/ Reading http://github.com/mongodb/mongo-python-driver

Copy the following to /usr/local/bin/load-lottery-tickets.pyimport random

from pymongo import Connection connection = Connection('appserver', 27017 )

db = connection.lottery collection = db.tickets

def buildDocument(): balls = [] count = 1 while ( count < 9 ): #populate results count = count + 1 ball = random.randint(1,48) balls.append(ball)

agent = random.randint(1,999) #agency number ticketnumber = random.randint(1,999999) #ticket number hashy = balls[1] * balls[2] * balls[3] * balls[4] * balls[5] * balls[6] * balls[7] * balls[0] document = {"lnhash":hashy, "week": wk, "agency": agent, "ticketnumber": ticketnumber, "luckynumbers": balls} return(document)

wk = 1while ( wk < 31 ): #loop through week 1 to week 30 count = 1 while ( count < 10001 ): #loop and insert 10000 documents per week document = buildDocument() collection.insert(document) count = count +1 wk = wk + 1 print wk

y = collection.count() print y

http://github.com/mongodb/mongo-python-driver

mailto:root@appserver

Load data

Run the python loading script. It will connect to the database and insert 300,000 documents. This simulates 30 weeks of Lottery games where 10,000 tickets were sold each week.[root@appserver /]> python2 /usr/local/bin/load-lottery-tickets.py

Check utilization

Lets check what is our disk utilization

week 30, with 300,000 records cmode81::> df -m Filesystem total used avail capacity Mounted on /vol/mongodb_shard1/ 3502MB 1617MB 1884MB 46% /vol/mongodb_shard1 /vol/mongodb_shard1/.snapshot 184MB 0MB 0MB 0% /vol/mongodb_shard1/.snapshot /vol/mongodb_shard2/ 3502MB 950MB 1967MB 33% /vol/mongodb_shard2 /vol/mongodb_shard2/.snapshot 184MB 0MB 146MB 0% /vol/mongodb_shard2/.snapshot

Disk usage is a matter of concern in MongoDB environments. While the balancer distributes chunks across different servers the size of these chunks can vary and lead to situations where one server is using more storage than another.

In some situations, the lack of free disk space can cause downtime.

This is difficult to address when servers are consuming local disk storage, but not really an issue on NetApp storage. As a best practice, enable volume autosize for all MongoDB FlexVols in order for these to automatically expand when they require disk space.

The balancer will then eventually kick in and redistribute chunks across shards if the number of chunks is considerably different. As of MongoDB version 2.0.2 the balancer won't balance chunks if one server is using 99% of its disk space, and the remaining servers 60%. This is because the balancer looks into the number of chunks in each server and not the total size vs the available space. For best practice allocate the same amount of disk space to all your servers.

Consider using NetApp Cluster-Mode and 64it aggregates, where the maximum FlexVol size is roughly 100TB.

SECTION: IMPACT OF STORAGE EFFICIENCY

This section shows the benefits of using NetApp C-Mode storage efficiency, and the savings in disk utilisation.

Enable storage efficiency

In this example, we chose to place all replicaset members storing the datafiles in the same FlexVol. This will bring us huge storage savings by simply enabling storage efficency (dedup and compression).

All ReplicaSet members contain the same data, since the Primary replicates to the secondaries, so each FlexVol will contain data that is high dedupable.

Check current state of dedup and compressioncmode81::> vol efficiency show -volume mongodb_shard1 (volume efficiency show) Vserver Volume State Status Progress ----------- ------------------- -------- ------------ ------------------- vserver1 mongodb_shard1 Disabled Idle Idle for 504:06:09

cmode81::> vol efficiency show -volume mongodb_shard2 (volume efficiency show) Vserver Volume State Status Progress ----------- ------------------- -------- ------------ ------------------- vserver1 mongodb_shard2 Disabled Idle Idle for 504:06:09

Enable Diagnostic modecmode81::> set diag

Warning: These diagnostic commands are for use by NetApp personnel only. Do you want to continue? {y|n}: y

Enable Volume efficiencycmode81::*> volume efficiency on -vserver vserver1 -volume mongodb_shard1 Efficiency for volume 'mongodb_shard1' of Vserver 'vserver1' is enabled. Already existing data could be processed by running "volume efficiency start -vserver vserver1 -volume mongodb_shard1 -scan-old-data true".

cmode81::*> volume efficiency on -vserver vserver1 -volume mongodb_shard2 Efficiency for volume 'mongodb_shard2' of Vserver 'vserver1' is enabled. Already existing data could be processed by running "volume efficiency start -vserver vserver1 -volume mongodb_shard2 -scan-old-data true".

Enable Compressioncmode81::*> volume efficiency modify -vserver vserver1 -volume mongodb_shard1 -compression true

cmode81::*> volume efficiency show -vserver vserver1 -volume mongodb_shard1 -fields compression vserver volume compression -------- -------------- ----------- vserver1 mongodb_shard1 true

cmode81::*> volume efficiency modify -vserver vserver1 -volume mongodb_shard2 -compression true

cmode81::*> volume efficiency show -vserver vserver1 -volume mongodb_shard2 -fields compression vserver volume compression -------- -------------- ----------- vserver1 mongodb_shard2 true

Scan existing blockscmode81::*> volume efficiency start -vserver vserver1 -volume mongodb_shard1 -scan-old-data true

Warning: The file system will be scanned to process existing data in volume 'mongodb_shard1' of Vserver 'vserver1'. This operation may initialize related existing metafiles. Do you want to continue? {y|n}: y The efficiency operation for volume 'mongodb_shard1' of Vserver 'vserver1' has started.

cmode81::*> volume efficiency start -vserver vserver1 -volume mongodb_shard2 -scan-old-data true

Warning: The file system will be scanned to process existing data in volume 'mongodb_shard2' of Vserver 'vserver1'. This operation may initialize related existing metafiles. Do you want to continue? {y|n}: y The efficiency operation for volume 'mongodb_shard2' of Vserver 'vserver1' has started.

Check for savingscmode81::*> vol efficiency stat (volume efficiency stat) Vserver Volume Allocated Saving %Saved ----------- ------------------- ------------------ ------------------ ------ vserver1 mongodb_shard1 617660 KB 1066476 KB 63% vserver1 mongodb_shard2 388304 KB 638924 KB 62%

cmode81::*> df -m Filesystem total used avail capacity Mounted on /vol/mongodb_shard1/ 3502MB 603MB 2898MB 17% /vol/mongodb_shard1 /vol/mongodb_shard1/.snapshot 184MB 0MB 184MB 0% /vol/mongodb_shard1/.snapshot /vol/mongodb_shard2/ 2918MB 379MB 2539MB 13% /vol/mongodb_shard2 /vol/mongodb_shard2/.snapshot 153MB 0MB 153MB 0% /vol/mongodb_shard2/.snapshot

cmode81::*> df -s Filesystem used saved %saved Vserver /vol/mongodb_shard1/ 617692 624608 50% vserver1 /vol/mongodb_shard2/ 388364 542060 58% vserver1

Example with 78 weeks loaded

Load additional data into the database collection, edit the python script that loads the lottery tickets and change the maximum week to 78wk = 31while ( wk < 79)#loop through week 31 to week 78 count = 1 while ( count < 10001 ): #loop and insert 10000 documents per week document = buildDocument() collection.insert(document) count = count +1 wk = wk + 1 print wk #collection.remove({})

Run it again:[root@appserver /]> python2 /usr/local/bin/load-lottery-tickets.py

Connect to the database

Check the number of documents in the collection:

mongos> db.tickets.count() 785718

Look up sharding informationmongos> db.printShardingStatus() --- Sharding Status --- sharding version: { "_id" : 1, "version" : 3 } shards: { "_id" : "shard1", "host" : "shard1/mongodb-shard1a:30011,mongodb-shard1a:27018,mongodb-shard1c:30013,mong odb-shard1b:27018" } { "_id" : "shard2", "host" : "shard2/mongodb-shard2a:30021,mongodb-shard2c:30023,mongodb-shard2b:30022" } databases: { "_id" : "admin", "partitioned" : false, "primary" : "config" } { "_id" : "test", "partitioned" : false, "primary" : "shard1" } { "_id" : "tickets", "partitioned" : true, "primary" : "shard2" } { "_id" : "lottery", "partitioned" : true, "primary" : "shard1" } lottery.tickets chunks: shard1 17 shard2 15 too many chunks to print, use verbose if you want to force print { "_id" : "settings", "partitioned" : false, "primary" : "shard2" }

Check disk usagecmode81::> df -h Filesystem total used avail capacity Mounted on

/vol/mongodb_shard1/ 2918MB 626MB 2291MB 21% /vol/mongodb_shard1 /vol/mongodb_shard1/.snapshot 153MB 0B 153MB 0% /vol/mongodb_shard1/.snapshot /vol/mongodb_shard2/ 2918MB 290MB 2627MB 10% /vol/mongodb_shard2 /vol/mongodb_shard2/.snapshot 153MB 0B 153MB 0% /vol/mongodb_shard2/.snapshot

Check disk savings:cmode81::> df -s Filesystem used saved %saved Vserver /vol/vol0/ 437464 0 0% cmode81-01 /vol/mongodb_shard1/ 641624 861624 57% vserver1 /vol/mongodb_shard2/ 297792 527988 64% vserver1

Storage Efficiency Savings Analysis

SECTION: CONSISTENT BACKUPS ACROSS MULTIPLE SERVERS

SnapCreator provides a framework that allows for taking consistent backups of applications. It is highly scalable and flexible.

Install SnapCreator

Login to the NOW website and download the SnapCreator software

Copy the downloaded file to the APPSERVER and extract it to /usr/local[root@appserver /]# cd /usr/local/scServer3.5.0[root@appserver scServer3.5.0]# ./snapcreator --profile setup

Do you accept the End User License Agreement (y|n): y

Setup NetApp Snap Creator Framework 3.5.0 Server (y|n): y

Enter serial number: 200000000

Enable GUI job monitor (Y|N): y

Enter job monitor size, how many jobs to allow: 10 Please Enter GUI Administrator Username: snapcreator Please Enter password for snapcreator: Please Confirm password for snapcreator:

INFO: Updated NetApp Snap Creator Framework 3.5.0 GUI

INFO: To start GUI please do the following:

cd /usr/local/scServer3.5.0/gui java -jar snapcreator.jar or java -jar snapcreator.jar -gui_port <gui_port>

INFO: To access NetApp Snap Creator Framework 3.5.0 GUI goto "http://appserver:8080" or "http://appserver:<gui_port>"

Create snapcreator user

SnapCreator requires a user account in the NetApp Cluster, cmode81::> security login create -username snapcreator -application ontapi -authmethod password -role vsadmin -vserver vserver1 Please enter a password for user 'snapcreator': Please enter it again:

cmode81::> security login show -vserver vserver1 Authentication Acct Vserver UserName Application Method Role Name Locked ----------- ---------------- ----------- -------------- ---------------- ------ vserver1 snapcreator ontapi password vsadmin no vserver1 vsadmin http password vsadmin yes vserver1 vsadmin ontapi password vsadmin yes vserver1 vsadmin ssh password vsadmin yes 4 entries were displayed.

Configure MongoDB backup

In our volume layout design for our shards and replicasets, every snapshot taken is always consistent across all replicasets as its an atomic operation. Due to the fact that we're leveraging one flexvol per Shard, this would be enough to have consistent backups of the MongoDB database.

In this document we'll expand this concept by following best-practice recommendations for MongoDB when backing up sharded databases. These recommendations include stopping the balancer, and locking the shards for writes while we take a snapshot.

This will provide a workable example of the capabilities of the SnapCreator framework to address consistency across diverse environments.

We'll configure SnapCreator to:

1. stop the balancer after all migrate operations have finished

2. perform a fsync and lock every shard for writes

3. take snapshots of all flexvols

4. unlock the databases for writes

5. restart the balancer

Copy python scripts to /usr/local/bin

fsync-lock.py

import random import time import sys from pymongo import Connection

server=sys.argv[1] port=int(sys.argv[2])

connection = Connection(server, port )

db = connection.admin #connect to database admin db.command("fsync", lock=True)

fsync-unlock.py import random import time import sys from pymongo import Connection

server=sys.argv[1] port=int(sys.argv[2]) connection = Connection(server, port )

db = connection.admin #connect to database admin db["$cmd.sys.unlock"].find_one()

wait-for-cleared-locks.py import randomimport timeimport sysfrom pymongo.son_manipulator import SONManipulatorfrom pymongo import Connection

server=sys.argv[1]port=int(sys.argv[2])

#connection = Connection('appserver', 27017 )connection = Connection(server, port )

class Transform(SONManipulator): def transform_incoming(self, son, collection): for (key, value) in son.items(): if isinstance(value, Custom): son[key] = encode_custom(value) elif isinstance(value, dict): # Make sure we recurse into sub-docs son[key] = self.transform_incoming(value, collection) return son

def transform_outgoing(self, son, collection): for (key, value) in son.items(): if isinstance(value, dict): if "_type" in value and value["_type"] == "custom": son[key] = decode_custom(value) else: # Again, make sure to recurse into sub-docs son[key] = self.transform_outgoing(value, collection) return son

db = connection.config #connect to database configdb.add_son_manipulator(Transform())

balancer_status = db.locks.find_one({'_id': 'balancer' })['state']print balancer_statusif balancer_status > 0 : #not null balancer_status = db.locks.find_one({'_id': 'balancer'})['state'] while ( balancer_status != 0 ): #wait until balancer has stopped time.sleep(1) balancer_status = db.locks.find_one({'_id': 'balancer'})['state'] print balancer_status

#STOP the balancerdb.settings.update({'_id': 'balancer'}, {'stopped': 'true'})

start-balancer.py import random import time import sys from pymongo import Connection

server=sys.argv[1] port=int(sys.argv[2])

connection = Connection(server, port )

db = connection.config #connect to database config db.settings.update({'_id': 'balancer'}, {'stopped': 'false'})#start balancer

Configure a SnapCreator profile for backup

Create a directory for the SnapCreator Profile.mkdir /usr/local/scServer3.5.0/configs/mongodb

copy master.conf from default.conf and change the following settings:########################### Required Options ###########################CONFIG_TYPE=STANDARDSNAME=MONGODBSNAP_TIMESTAMP_ONLY=NVOLUMES=vserver1-mgmt:mongodb_shard1,mongodb_shard2NTAP_SNAPSHOT_RETENTIONS=daily:2NTAP_USERS=vserver1-mgmt:snapcreator/netapp123NTAP_PWD_PROTECTION=NTRANSPORT=HTTPPORT=80LOG_NUM=30LOG_TRACE_ENABLE=YSC_TMP_DIR=

######################### CMODE Settings #########################CMODE_CLUSTER_USERS=cmode81a-cluster-mgmt:admin/netapp123CMODE_CLUSTER_NAME=cmode81a-cluster-mgmtCMODE_SNAPSHOT_FORCE_DELETE=Y

########################### Snapshot Options ###########################NTAP_SNAPSHOT_RETENTION_AGE=2SNAPDRIVE=NSNAPDRIVE_DISCOVERY=NNTAP_SNAPSHOT_DISABLE=NNTAP_SNAPSHOT_NODELETE=NNTAP_CONSISTENCY_GROUP_SNAPSHOT=NNTAP_CONSISTENCY_GROUP_TIMEOUT=mediumNTAP_CONSISTENCY_GROUP_WAFL_SYNC=NNTAP_SNAPSHOT_DELETE_BY_AGE_ONLY=NNTAP_SNAPSHOT_DEPENDENCY_IGNORE=NNTAP_SNAPSHOT_RESTORE_AUTO_DETECT=N

########################## Cloning Options ##########################NTAP_VOL_CLONE_RESERVE=noneNTAP_LUN_CLONE_RESERVATION=falseNTAP_CLONE_IGROUP_MAP=NTAP_CLONE_FOR_BACKUP=YNTAP_CLONE_SECONDARY=NNTAP_CLONE_SECONDARY_VOLUMES=NTAP_NUM_VOL_CLONES=1NTAP_NFS_EXPORT_HOST=mongodb-clonedbNTAP_NFS_EXPORT_ACCESS=rootNTAP_NFS_EXPORT_PERSISTENT=false

####################### APP Commands #######################ARCHIVE_CMD01=MOUNT_CMD01=UMOUNT_CMD01=

####################### Pre Commands #######################APP_QUIESCE_CMD01=/usr/bin/python2.6 /usr/local/bin/wait-for-cleared-locks.py appserver 27017APP_QUIESCE_CMD02=/usr/bin/python2.6 /usr/local/bin/fsync-lock.py mongodb-shard1a 30011APP_QUIESCE_CMD03=/usr/bin/python2.6 /usr/local/bin/fsync-lock.py mongodb-shard1b 30012APP_QUIESCE_CMD04=/usr/bin/python2.6 /usr/local/bin/fsync-lock.py mongodb-shard1c 30013APP_QUIESCE_CMD05=/usr/bin/python2.6 /usr/local/bin/fsync-lock.py mongodb-shard2a 30021APP_QUIESCE_CMD06=/usr/bin/python2.6 /usr/local/bin/fsync-lock.py mongodb-shard2b 30022

APP_QUIESCE_CMD07=/usr/bin/python2.6 /usr/local/bin/fsync-lock.py mongodb-shard2c 30023PRE_NTAP_CMD01=PRE_NTAP_CLONE_DELETE_CMD01=PRE_APP_UNQUIESCE_CMD01=PRE_RESTORE_CMD01=PRE_CLONE_CREATE_CMD01=

######################## Post Commands ########################POST_APP_QUIESCE_CMD01=PRE_EXIT_CMD01=/usr/bin/python2.6 /usr/local/bin/fsync-unlock.py mongodb-shard1a 30011PRE_EXIT_CMD02=/usr/bin/python2.6 /usr/local/bin/fsync-unlock.py mongodb-shard1b 30012PRE_EXIT_CMD03=/usr/bin/python2.6 /usr/local/bin/fsync-unlock.py mongodb-shard1c 30013PRE_EXIT_CMD04=/usr/bin/python2.6 /usr/local/bin/fsync-unlock.py mongodb-shard2a 30021PRE_EXIT_CMD05=/usr/bin/python2.6 /usr/local/bin/fsync-unlock.py mongodb-shard2b 30022PRE_EXIT_CMD06=/usr/bin/python2.6 /usr/local/bin/fsync-unlock.py mongodb-shard2c 30023PRE_EXIT_CMD07=/usr/bin/python2.6 /usr/local/bin/start-balancer.py appserver 27017POST_NTAP_DATA_TRANSFER_CMD01=APP_UNQUIESCE_CMD01=/usr/bin/python2.6 /usr/local/bin/fsync-unlock.py mongodb-shard1a 30011APP_UNQUIESCE_CMD02=/usr/bin/python2.6 /usr/local/bin/fsync-unlock.py mongodb-shard1b 30012APP_UNQUIESCE_CMD03=/usr/bin/python2.6 /usr/local/bin/fsync-unlock.py mongodb-shard1c 30013APP_UNQUIESCE_CMD04=/usr/bin/python2.6 /usr/local/bin/fsync-unlock.py mongodb-shard2a 30021APP_UNQUIESCE_CMD05=/usr/bin/python2.6 /usr/local/bin/fsync-unlock.py mongodb-shard2b 30022APP_UNQUIESCE_CMD06=/usr/bin/python2.6 /usr/local/bin/fsync-unlock.py mongodb-shard2c 30023APP_UNQUIESCE_CMD07=/usr/bin/python2.6 /usr/local/bin/start-balancer.py appserver 27017POST_RESTORE_CMD01=POST_CLONE_CREATE_CMD01=

SC_AGENT=SC_AGENT_TIMEOUT=60SC_AGENT_UNQUIESCE_TIMEOUT=SC_CLONE_TARGET=SC_AGENT_WATCHDOG_ENABLE=YSC_AGENT_LOG_ENABLE=Y

Take a snapcreator backup [root@appserver mongodb]# /usr/local/scServer3.5.0/snapcreator --profile mongodb --config master --action snap --policy daily --verbose

[Tue Feb 21 21:25:57 2012] INFO: Logfile timestamp: 20120221212557[Tue Feb 21 21:25:57 2012] INFO: Removing log master.out.20120221064158.log[Tue Feb 21 21:25:57 2012] INFO: Removing log master.debug.20120221064158.log

########## Parsing Environment Parameters ##########

########## PRE APPLICATION QUIESCE COMMANDS ##########[Tue Feb 21 21:25:57 2012] INFO: No commands defined

########## PRE APPLICATION QUIESCE COMMANDS FINISHED SUCCESSFULLY ##########

########## APPLICATION QUIESCE COMMANDS ##########

########## Detecting Data OnTap mode for cmode81a-cluster-mgmt ##########[Tue Feb 21 21:25:57 2012] INFO: Data OnTap Cluster mode detected[Tue Feb 21 21:25:57 2012] INFO: Running application quiesce command APP_QUIESCE_CMD01 [/usr/bin/python2.6 /usr/local/bin/wait-for-cleared-locks.py appserver 27017][Tue Feb 21 21:25:58 2012] INFO: Running application quiesce command [/usr/bin/python2.6 /usr/local/bin/wait-for-cleared-locks.py appserver 27017] finished successfully

########## Detecting Data OnTap mode for cmode81a-cluster-mgmt ##########[Tue Feb 21 21:25:58 2012] INFO: Data OnTap Cluster mode detected[Tue Feb 21 21:25:58 2012] INFO: Running application quiesce command APP_QUIESCE_CMD02 [/usr/bin/python2.6 /usr/local/bin/fsync-lock.py mongodb-shard1a 30011][Tue Feb 21 21:25:58 2012] INFO: Running application quiesce command [/usr/bin/python2.6 /usr/local/bin/fsync-lock.py mongodb-shard1a 30011] finished successfully

########## Detecting Data OnTap mode for cmode81a-cluster-mgmt ##########[Tue Feb 21 21:25:58 2012] INFO: Data OnTap Cluster mode detected[Tue Feb 21 21:25:58 2012] INFO: Running application quiesce command APP_QUIESCE_CMD03 [/usr/bin/python2.6 /usr/local/bin/fsync-lock.py mongodb-shard1b 30012][Tue Feb 21 21:25:59 2012] INFO: Running application quiesce command [/usr/bin/python2.6 /usr/local/bin/fsync-lock.py mongodb-shard1b 30012] finished successfully

########## Detecting Data OnTap mode for cmode81a-cluster-mgmt ##########[Tue Feb 21 21:25:59 2012] INFO: Data OnTap Cluster mode detected

[Tue Feb 21 21:25:59 2012] INFO: Running application quiesce command APP_QUIESCE_CMD04 [/usr/bin/python2.6 /usr/local/bin/fsync-lock.py mongodb-shard1c 30013][Tue Feb 21 21:25:59 2012] INFO: Running application quiesce command [/usr/bin/python2.6 /usr/local/bin/fsync-lock.py mongodb-shard1c 30013] finished successfully

########## Detecting Data OnTap mode for cmode81a-cluster-mgmt ##########[Tue Feb 21 21:25:59 2012] INFO: Data OnTap Cluster mode detected[Tue Feb 21 21:25:59 2012] INFO: Running application quiesce command APP_QUIESCE_CMD05 [/usr/bin/python2.6 /usr/local/bin/fsync-lock.py mongodb-shard2a 30021][Tue Feb 21 21:26:00 2012] INFO: Running application quiesce command [/usr/bin/python2.6 /usr/local/bin/fsync-lock.py mongodb-shard2a 30021] finished successfully

########## Detecting Data OnTap mode for cmode81a-cluster-mgmt ##########[Tue Feb 21 21:26:00 2012] INFO: Data OnTap Cluster mode detected[Tue Feb 21 21:26:00 2012] INFO: Running application quiesce command APP_QUIESCE_CMD06 [/usr/bin/python2.6 /usr/local/bin/fsync-lock.py mongodb-shard2b 30022][Tue Feb 21 21:26:00 2012] INFO: Running application quiesce command [/usr/bin/python2.6 /usr/local/bin/fsync-lock.py mongodb-shard2b 30022] finished successfully

########## Detecting Data OnTap mode for cmode81a-cluster-mgmt ##########[Tue Feb 21 21:26:00 2012] INFO: Data OnTap Cluster mode detected[Tue Feb 21 21:26:00 2012] INFO: Running application quiesce command APP_QUIESCE_CMD07 [/usr/bin/python2.6 /usr/local/bin/fsync-lock.py mongodb-shard2c 30023][Tue Feb 21 21:26:01 2012] INFO: Running application quiesce command [/usr/bin/python2.6 /usr/local/bin/fsync-lock.py mongodb-shard2c 30023] finished successfully

########## APPLICATION QUIESCE COMMANDS FINISHED SUCCESSFULLY ##########

########## POST APPLICATION QUIESCE COMMANDS ##########[Tue Feb 21 21:26:01 2012] INFO: No commands defined

########## POST APPLICATION QUIESCE COMMANDS FINISHED SUCCESSFULLY ##########

########## PRE NETAPP COMMANDS ##########[Tue Feb 21 21:26:01 2012] INFO: No commands defined

########## PRE NETAPP COMMANDS FINISHED SUCCESSFULLY ##########


########## Detecting Data OnTap mode for vserver1-mgmt ##########[Tue Feb 21 21:26:01 2012] INFO: Data OnTap Cluster mode detected

########## Detecting Data OnTap mode for cmode81a-cluster-mgmt ##########[Tue Feb 21 21:26:01 2012] INFO: Data OnTap Cluster mode detected[Tue Feb 21 21:26:01 2012] INFO: Discover cmode cluster nodes on cmode81a-cluster-mgmt[Tue Feb 21 21:26:01 2012] INFO: Discover cmode cluster nodes on cmode81a-cluster-mgmt completed successfully

########## Generating Info ASUP on vserver1-mgmt ##########[Tue Feb 21 21:26:02 2012] INFO: NetApp ASUP create on cmode81a-cluster-mgmt:cmode81-01 finished successfully

########## Gathering Information for vserver1-mgmt:mongodb_shard1 ##########[Tue Feb 21 21:26:02 2012] INFO: Performing NetApp Snapshot Inventory for mongodb_shard1 on vserver1-mgmt[Tue Feb 21 21:26:02 2012] INFO: NetApp Snapshot Inventory of mongodb_shard1 on vserver1-mgmt completed Successfully


########## Running NetApp Snapshot Rename on Primary vserver1-mgmt ##########[Tue Feb 21 21:26:03 2012] INFO: Rename the recent snapshot from MONGODB-daily_recent to MONGODB-daily_20120301135116 for vserver1-mgmt:mongodb_shard1[Tue Feb 21 21:26:04 2012] INFO: NetApp Snapshot Rename for mongodb_shard1 on vserver1-mgmt completed Successfully[Tue Feb 21 21:26:04 2012] INFO: Rename the recent snapshot from MONGODB-daily_recent to MONGODB-daily_20120301135117 for vserver1-mgmt:mongodb_shard2[Tue Feb 21 21:26:05 2012] INFO: NetApp Snapshot Rename for mongodb_shard2 on vserver1-mgmt completed Successfully

########## Creating snapshot(s) ##########[Tue Feb 21 21:26:05 2012] INFO: NetApp Snap Creator Framework 3.5.0 detected that SnapDrive is not

being used. File system consistency cannot be guaranteed for SAN/iSAN environments

########## Taking Snapshot on Primary vserver1-mgmt:mongodb_shard1 ##########[Tue Feb 21 21:26:05 2012] INFO: Creating NetApp Snapshot for mongodb_shard1 on vserver1-mgmt[Tue Feb 21 21:26:07 2012] INFO: NetApp Snapshot Create of MONGODB-daily_recent on vserver1-mgmt:mongodb_shard1 Completed Successfully

########## Taking Snapshot on Primary vserver1-mgmt:mongodb_shard2 ##########[Tue Feb 21 21:26:07 2012] INFO: Creating NetApp Snapshot for mongodb_shard2 on vserver1-mgmt[Tue Feb 21 21:26:09 2012] INFO: NetApp Snapshot Create of MONGODB-daily_recent on vserver1-mgmt:mongodb_shard2 Completed Successfully

########## PRE APPLICATION UNQUIESCE COMMANDS ##########[Tue Feb 21 21:26:11 2012] INFO: No commands defined

########## PRE APPLICATION UNQUIESCE COMMANDS FINISHED SUCCESSFULLY ##########

########## APPLICATION UNQUIESCE COMMANDS ##########

########## Detecting Data OnTap mode for cmode81a-cluster-mgmt ##########[Tue Feb 21 21:26:12 2012] INFO: Data OnTap Cluster mode detected[Tue Feb 21 21:26:12 2012] INFO: Running application unquiesce command APP_UNQUIESCE_CMD01 [/usr/bin/python2.6 /usr/local/bin/fsync-unlock.py mongodb-shard1a 30011][Tue Feb 21 21:26:12 2012] INFO: Running application unquiesce command [/usr/bin/python2.6 /usr/local/bin/fsync-unlock.py mongodb-shard1a 30011] finished successfully

########## Detecting Data OnTap mode for cmode81a-cluster-mgmt ##########[Tue Feb 21 21:26:12 2012] INFO: Data OnTap Cluster mode detected[Tue Feb 21 21:26:12 2012] INFO: Running application unquiesce command APP_UNQUIESCE_CMD02 [/usr/bin/python2.6 /usr/local/bin/fsync-unlock.py mongodb-shard1b 30012][Tue Feb 21 21:26:12 2012] INFO: Running application unquiesce command [/usr/bin/python2.6 /usr/local/bin/fsync-unlock.py mongodb-shard1b 30012] finished successfully

########## Detecting Data OnTap mode for cmode81a-cluster-mgmt ##########[Tue Feb 21 21:26:13 2012] INFO: Data OnTap Cluster mode detected[Tue Feb 21 21:26:13 2012] INFO: Running application unquiesce command APP_UNQUIESCE_CMD03 [/usr/bin/python2.6 /usr/local/bin/fsync-unlock.py mongodb-shard1c 30013][Tue Feb 21 21:26:13 2012] INFO: Running application unquiesce command [/usr/bin/python2.6 /usr/local/bin/fsync-unlock.py mongodb-shard1c 30013] finished successfully

########## Detecting Data OnTap mode for cmode81a-cluster-mgmt ##########[Tue Feb 21 21:26:13 2012] INFO: Data OnTap Cluster mode detected[Tue Feb 21 21:26:13 2012] INFO: Running application unquiesce command APP_UNQUIESCE_CMD04 [/usr/bin/python2.6 /usr/local/bin/fsync-unlock.py mongodb-shard2a 30021][Tue Feb 21 21:26:13 2012] INFO: Running application unquiesce command [/usr/bin/python2.6 /usr/local/bin/fsync-unlock.py mongodb-shard2a 30021] finished successfully

########## Detecting Data OnTap mode for cmode81a-cluster-mgmt ##########[Tue Feb 21 21:26:13 2012] INFO: Data OnTap Cluster mode detected[Tue Feb 21 21:26:13 2012] INFO: Running application unquiesce command APP_UNQUIESCE_CMD05 [/usr/bin/python2.6 /usr/local/bin/fsync-unlock.py mongodb-shard2b 30022][Tue Feb 21 21:26:14 2012] INFO: Running application unquiesce command [/usr/bin/python2.6 /usr/local/bin/fsync-unlock.py mongodb-shard2b 30022] finished successfully

########## Detecting Data OnTap mode for cmode81a-cluster-mgmt ##########[Tue Feb 21 21:26:14 2012] INFO: Data OnTap Cluster mode detected[Tue Feb 21 21:26:14 2012] INFO: Running application unquiesce command APP_UNQUIESCE_CMD06 [/usr/bin/python2.6 /usr/local/bin/fsync-unlock.py mongodb-shard2c 30023][Tue Feb 21 21:26:14 2012] INFO: Running application unquiesce command [/usr/bin/python2.6 /usr/local/bin/fsync-unlock.py mongodb-shard2c 30023] finished successfully

########## Detecting Data OnTap mode for cmode81a-cluster-mgmt ##########[Tue Feb 21 21:26:14 2012] INFO: Data OnTap Cluster mode detected[Tue Feb 21 21:26:14 2012] INFO: Running application unquiesce command APP_UNQUIESCE_CMD07 [/usr/bin/python2.6 /usr/local/bin/start-balancer.py appserver 27017][Tue Feb 21 21:26:15 2012] INFO: Running application unquiesce command [/usr/bin/python2.6 /usr/local/bin/start-balancer.py appserver 27017] finished successfully

########## APPLICATION UNQUIESCE COMMANDS FINISHED SUCCESSFULLY ##########

########## POST APPLICATION UNQUIESCE COMMANDS ##########[Tue Feb 21 21:26:15 2012] INFO: No commands defined

########## POST APPLICATION UNQUIESCE COMMANDS FINISHED SUCCESSFULLY ##########

########## Generating Info ASUP on vserver1-mgmt ##########[Tue Feb 21 21:26:15 2012] INFO: NetApp ASUP create on cmode81a-cluster-mgmt:cmode81-01 finished

successfully

########## POST NETAPP DATA TRANSFER COMMANDS ##########[Tue Feb 21 21:26:15 2012] INFO: No commands defined

########## POST NETAPP DATA TRANSFER COMMANDS FINISHED SUCCESSFULLY ##########

########## Running NetApp Snapshot Delete on Primary vserver1-mgmt ##########[Tue Feb 21 21:26:30 2012] WARN: WARN: More than 2 snapshots exist (7) but they could not be deleted because they are not older than the retention age (2 days) for vserver1-mgmt:mongodb_shard1[Tue Feb 21 21:26:30 2012] WARN: WARN: More than 2 snapshots exist (7) but they could not be deleted because they are not older than the retention age (2 days) for vserver1-mgmt:mongodb_shard2

########## POST NETAPP COMMANDS ##########[Tue Feb 21 21:26:30 2012] INFO: No commands defined

########## POST NETAPP COMMANDS FINISHED SUCCESSFULLY ##########

########## ARCHIVE COMMANDS ##########[Tue Feb 21 21:26:30 2012] INFO: No commands defined

########## ARCHIVE COMMANDS FINISHED SUCCESSFULLY ##########

########## NetApp Snap Creator Framework 3.5.0 finished successfully ##########[Tue Feb 21 21:26:30 2012] INFO: INFO: Snap Creator finished successfully ( Action: snap )

We can see from the logfiles that this backup took exactly 33 seconds:

[Tue Feb 21 21:25:57 2012] INFO: Logfile timestamp: 20120221212557[Tue Feb 21 21:26:30 2012] INFO: INFO: Snap Creator finished successfully ( Action: snap )

List existing snapcreator snapshots

Use the following command to list SnapCreator backups

[root@appserver scServer3.5.0]# /usr/local/scServer3.5.0/snapcreator --profile mongodb --config master --action snaplist

########## NetApp Snap Creator Framework 3.5.0 Snapshot (Primary) List for vserver1-mgmt:mongodb_shard1 #########

### Snapshot Name ### ### Snapshot Timestamp ### MONGODB-daily_20120228134412 Feb 28 2012 13:44:12 MONGODB-daily_20120228135612 Feb 28 2012 13:56:12 MONGODB-daily_20120228152346 Feb 28 2012 15:23:46 MONGODB-daily_20120228152757 Feb 28 2012 15:27:57 MONGODB-daily_20120228163417 Feb 28 2012 16:34:17 MONGODB-daily_20120229135802 Feb 29 2012 13:58:02 MONGODB-daily_20120229140023 Feb 29 2012 14:00:23 MONGODB-daily_20120301135116 Mar 01 2012 13:51:16 MONGODB-daily_recent Mar 01 2012 14:48:04



SECTION: DATABASE CLONING FOR DEV/TEST ENVIRONMENTS

Pre-production or development environments can be built of an existing production snapshot backup; the snapshoted volumes can be presented to any clone host through the use of NetApp flexclones.This allows for quickly building DEV/TEST environments without having to double the amount of storage required. FlexClones are a fast, scalableand efficient way of providing DEV/TEST environments on demand.

In this example, we will be cloning all databases across all shard servers into a single server named mongodb-clonedb.

FIX /ETC/HOSTS

MongoDB has a limitation with regard to sharded database clones; the configuration of the database is kept in the configdb which is located on the config servers /data/configdb.

This configdb contains references to the hostname:ports that make each shard of the database. This causes a problem during cloning, as the diverse config servers and database servers will try to connect to the production servers as soon the processes are started. In order to avoid this, we need to fix etc/hosts so that the hostnames of the different shard members point to the mongodb-clonedb ip address.

LOGIN TO MONGODB-CLONEDB

Append entries to all servers that make this database, add those entries as aliases to the mongodb-clonedb IP address.

[root@mongodb-clonedb /]# cat /etc/hosts ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 127.0.0.1 localhost.localdomain localhost localhost4.localdomain4 localhost4 # Auto-generated hostname. Please do not remove this comment. 172.16.1.101 mongodb-clonedb mongodb-clonedb mongodb-shard1a mongodb-shard1a mongodb-shard1b mongodb-shard1b mongodb-shard1c mongodb-shard1c mongodb-shard2a mongodb-shard2a mongodb-shard2b mongodb-shard2b mongodb-shard2c mongodb-shard2c

Install SnapCreator AGENT on the clonedb server[root@mongodb-clonedb /]# cd /usr/local/scServer3.5.0[root@mongodb-clonedb scServer3.5.0]# ./snapcreator --profile setup

Copy and extract the software to /usr/local

Base Software EULA v-11-02-11


Setup NetApp Snap Creator Framework 3.5.0 Server (y|n): n

Setup NetApp Snap Creator Framework 3.5.0 Agent (y|n): y

INFO: Updated NetApp Snap Creator Framework 3.5.0 Agent INFO: To start the NetApp Snap Creator Framework 3.5.0 Agent run "/usr/local/scAgent3.5.0/bin/scAgent start" INFO: To stop the NetApp Snap Creator Framework 3.5.0 Agent run "/usr/local/scAgent3.5.0/bin/scAgent stop" [root@mongodb-clonedb scAgent3.5.0]#

Start the SnapCreator Agent[root@mongodb-clonedb scAgent3.5.0]# /usr/local/scAgent3.5.0/bin/scAgent start Starting scAgent:

Checking Status of scAgent: Running [root@mongodb-clonedb scAgent3.5.0]#

Configure SC Agent access

[root@mongodb-clonedb scAgent3.5.0]# cat /usr/local/scAgent3.5.0/config/agent.conf ############################################ ### SnapCreator Agent Configuration File ### ############################################ ###################################################################### ### Command to allow or wildcard "*" can be used to allow anything ### ###################################################################### host: scServer@* command: *

Prepare host environment

Create the MongoDB user[root@mongodb-clonedb /]# groupadd -g 8000 mongodb [root@mongodb-clonedb /]# useradd -u 8000 -g 8000 -d /data mongodb [root@mongodb-clonedb /]#

Create directories for all the cloned mongodb instancesmkdir -p /data/mongodb-shard1amkdir -p /data/mongodb-shard1bmkdir -p /data/mongodb-shard1cmkdir -p /data/mongodb-shard2amkdir -p /data/mongodb-shard2bmkdir -p /data/mongodb-shard2c

chmod -R 700 /data chown -R mongodb:mongodb /data

Build SnapCreator Profiles

LOGIN TO appserver

Copy SnapCreator default.conf to mongodb/clonedb.conf and edit clonedb.conf

########################### Required Options ###########################CONFIG_TYPE=STANDARDSNAME=MONGODBSNAP_TIMESTAMP_ONLY=NVOLUMES=vserver1-mgmt:mongodb_shard1,mongodb_shard2NTAP_SNAPSHOT_RETENTIONS=daily:2NTAP_USERS=vserver1-mgmt:snapcreator/netapp123NTAP_PWD_PROTECTION=NTRANSPORT=HTTPPORT=80LOG_NUM=30LOG_TRACE_ENABLE=YSC_TMP_DIR=


########################### Snapshot Options ###########################NTAP_SNAPSHOT_RETENTION_AGE=SNAPDRIVE=NSNAPDRIVE_DISCOVERY=N

NTAP_SNAPSHOT_DISABLE=YNTAP_SNAPSHOT_NODELETE=YNTAP_CONSISTENCY_GROUP_SNAPSHOT=NNTAP_CONSISTENCY_GROUP_TIMEOUT=mediumNTAP_CONSISTENCY_GROUP_WAFL_SYNC=NNTAP_SNAPSHOT_DELETE_BY_AGE_ONLY=NNTAP_SNAPSHOT_DEPENDENCY_IGNORE=NNTAP_SNAPSHOT_RESTORE_AUTO_DETECT=N

########################## Cloning Options ##########################NTAP_VOL_CLONE_RESERVE=noneNTAP_LUN_CLONE_RESERVATION=falseNTAP_CLONE_IGROUP_MAP=NTAP_CLONE_FOR_BACKUP=YNTAP_CLONE_SECONDARY=NNTAP_CLONE_SECONDARY_VOLUMES=NTAP_NUM_VOL_CLONES=1NTAP_NFS_EXPORT_HOST=mongodb-clonedbNTAP_NFS_EXPORT_ACCESS=read-writeNTAP_NFS_EXPORT_PERSISTENT=false

####################### APP Commands #######################ARCHIVE_CMD01=UMOUNT_CMD01=AGENT:killall -u mongodb ; exit 0UMOUNT_CMD02=AGENT:umount /data/mongodb-shard* ; exit 0MOUNT_CMD01=AGENT:mount -t nfs -o rw,vers=3,rsize=65535,wsize=65535,hard,proto=tcp,timeo=600,retrans=2,sec=sys,addr=172.16.1.101 vserver1-nfs:/cl_clonedb_mongodb_shard1_%SNAP_TIME/mongodb_shard1a /data/mongodb-shard1aMOUNT_CMD02=AGENT:mount -t nfs -o rw,vers=3,rsize=65535,wsize=65535,hard,proto=tcp,timeo=600,retrans=2,sec=sys,addr=172.16.1.101 vserver1-nfs:/cl_clonedb_mongodb_shard1_%SNAP_TIME/mongodb_shard1b /data/mongodb-shard1bMOUNT_CMD03=AGENT:mount -t nfs -o rw,vers=3,rsize=65535,wsize=65535,hard,proto=tcp,timeo=600,retrans=2,sec=sys,addr=172.16.1.101 vserver1-nfs:/cl_clonedb_mongodb_shard1_%SNAP_TIME/mongodb_shard1c /data/mongodb-shard1cMOUNT_CMD04=AGENT:mount -t nfs -o rw,vers=3,rsize=65535,wsize=65535,hard,proto=tcp,timeo=600,retrans=2,sec=sys,addr=172.16.1.101 vserver1-nfs:/cl_clonedb_mongodb_shard2_%SNAP_TIME/mongodb_shard2a /data/mongodb-shard2aMOUNT_CMD05=AGENT:mount -t nfs -o rw,vers=3,rsize=65535,wsize=65535,hard,proto=tcp,timeo=600,retrans=2,sec=sys,addr=172.16.1.101 vserver1-nfs:/cl_clonedb_mongodb_shard2_%SNAP_TIME/mongodb_shard2b /data/mongodb-shard2bMOUNT_CMD06=AGENT:mount -t nfs -o rw,vers=3,rsize=65535,wsize=65535,hard,proto=tcp,timeo=600,retrans=2,sec=sys,addr=172.16.1.101 vserver1-nfs:/cl_clonedb_mongodb_shard2_%SNAP_TIME/mongodb_shard2c /data/mongodb-shard2cMOUNT_CMD07=AGENT:chown -R mongodb:mongodb /data ; exit 0

MOUNT_CMD08=AGENT:rm -f /data/mongodb-shard*/db/mongod.lock

MOUNT_CMD09=AGENT:su - mongodb -c '/opt/mongodb/bin/mongod --replSet shard1/mongodb-shard1b:30012,mongodb-shard1c:30013 --journal --rest --shardsvr --port 30011 --dbpath /data/mongodb-shard1a/db --fork --logpath /data/mongodb-shard1a/db/mongod.log'MOUNT_CMD10=AGENT:su - mongodb -c '/opt/mongodb/bin/mongod --replSet shard1/mongodb-shard1a:30011,mongodb-shard1c:30013 --journal --rest --shardsvr --port 30012 --dbpath /data/mongodb-shard1b/db --fork --logpath /data/mongodb-shard1b/db/mongod.log'MOUNT_CMD11=AGENT:su - mongodb -c '/opt/mongodb/bin/mongod --replSet shard1/mongodb-shard1a:30011,mongodb-shard1b:30012 --journal --rest --shardsvr --port 30013 --dbpath /data/mongodb-shard1c/db --fork --logpath /data/mongodb-shard1c/db/mongod.log'

MOUNT_CMD12=AGENT:su - mongodb -c '/opt/mongodb/bin/mongod --replSet shard2/mongodb-shard2b:30022,mongodb-shard2c:30023 --journal --rest --shardsvr --port 30021 --dbpath /data/mongodb-shard2a/db --fork --logpath /data/mongodb-shard2a/db/mongod.log'MOUNT_CMD13=AGENT:su - mongodb -c '/opt/mongodb/bin/mongod --replSet shard2/mongodb-shard2a:30021,mongodb-shard2c:30023 --journal --rest --shardsvr --port 30022 --dbpath /data/mongodb-shard2b/db --fork --logpath /data/mongodb-shard2b/db/mongod.log'MOUNT_CMD14=AGENT:su - mongodb -c '/opt/mongodb/bin/mongod --replSet shard2/mongodb-shard2a:30021,mongodb-shard2b:30022 --journal --rest --shardsvr --port 30023 --dbpath /data/mongodb-shard2c/db --fork --logpath /data/mongodb-shard2c/db/mongod.log'

MOUNT_CMD15=AGENT:su - mongodb -c '/opt/mongodb/bin/mongod --configsvr --port 4011 --dbpath /data/mongodb-shard1a/configdb/ --fork --logpath /data/mongodb-shard1a/configdb/mongod.log'MOUNT_CMD16=AGENT:su - mongodb -c '/opt/mongodb/bin/mongod --configsvr --port 4021 --dbpath /data/mongodb-shard2a/configdb/ --fork --logpath /data/mongodb-shard2a/configdb/mongod.log'MOUNT_CMD17=AGENT:su - mongodb -c '/opt/mongodb/bin/mongod --configsvr --port 4023 --dbpath /data/mongodb-shard2c/configdb/ --fork --logpath /data/mongodb-shard2c/configdb/mongod.log'

MOUNT_CMD19=AGENT:su - mongodb -c 'sleep 60; /opt/mongodb/bin/mongos --configdb mongodb-clonedb:4011,mongodb-clonedb:4021,mongodb-clonedb:4023 --fork --logpath /data/mongos.log'

####################### Pre Commands #######################PRE_APP_QUIESCE_CMD01=PRE_NTAP_CMD01=PRE_NTAP_CLONE_DELETE_CMD01=PRE_APP_UNQUIESCE_CMD01=PRE_RESTORE_CMD01=PRE_CLONE_CREATE_CMD01=

######################## Post Commands ########################POST_APP_QUIESCE_CMD01=PRE_EXIT_CMD01=

POST_NTAP_DATA_TRANSFER_CMD01=POST_APP_UNQUIESCE_CMD01=POST_RESTORE_CMD01=POST_CLONE_CREATE_CMD01=

SC_AGENT=mongodb-clonedb:9090SC_AGENT_TIMEOUT=120SC_AGENT_UNQUIESCE_TIMEOUT=SC_CLONE_TARGET=mongodb-clonedb:9090SC_AGENT_WATCHDOG_ENABLE=YSC_AGENT_LOG_ENABLE=Y

Build SnapCreator Clone From Snapshot [root@appserver scServer3.5.0]# /usr/local/scServer3.5.0/snapcreator --profile mongodb --config clonedb --action clone_vol --snap_name MONGODB-daily_recent --policy daily --clone_name mongodb_clonedb –verbose

[Tue Feb 21 21:34:00 2012] INFO: Logfile timestamp: 20120221213400[Tue Feb 21 21:34:00 2012] INFO: Removing log clonedb.out.20120221142317.log[Tue Feb 21 21:34:00 2012] INFO: Removing log clonedb.debug.20120221142317.log[Tue Feb 21 21:34:00 2012] INFO: Removing log clonedb.stderr.20120221142317.log


########## PRE APPLICATION QUIESCE COMMANDS ##########[Tue Feb 21 21:34:00 2012] INFO: No commands defined

########## PRE APPLICATION QUIESCE COMMANDS FINISHED SUCCESSFULLY ##########

########## POST APPLICATION QUIESCE COMMANDS ##########[Tue Feb 21 21:34:00 2012] INFO: No commands defined

########## POST APPLICATION QUIESCE COMMANDS FINISHED SUCCESSFULLY ##########

########## PRE NETAPP COMMANDS ##########[Tue Feb 21 21:34:00 2012] INFO: No commands defined

########## PRE NETAPP COMMANDS FINISHED SUCCESSFULLY ##########

########## Parsing Environment Parameters ##########[Tue Feb 21 21:34:00 2012] WARN: Snapshot's will not be deleted, if this is not desired please set NTAP_SNAPSHOT_NODELETE=N in config file

########## Detecting Data OnTap mode for vserver1-mgmt ##########[Tue Feb 21 21:34:01 2012] INFO: Data OnTap Cluster mode detected

########## Detecting Data OnTap mode for cmode81a-cluster-mgmt ##########[Tue Feb 21 21:34:01 2012] INFO: Data OnTap Cluster mode detected[Tue Feb 21 21:34:01 2012] INFO: Discover cmode cluster nodes on cmode81a-cluster-mgmt[Tue Feb 21 21:34:01 2012] INFO: Discover cmode cluster nodes on cmode81a-cluster-mgmt completed successfully


########## Gathering Information for luns on vserver1-mgmt ##########



########## PRE APPLICATION UNQUIESCE COMMANDS ##########[Tue Feb 21 21:34:05 2012] INFO: No commands defined

########## PRE APPLICATION UNQUIESCE COMMANDS FINISHED SUCCESSFULLY ##########

########## POST APPLICATION UNQUIESCE COMMANDS ##########[Tue Feb 21 21:34:05 2012] INFO: No commands defined

########## POST APPLICATION UNQUIESCE COMMANDS FINISHED SUCCESSFULLY ##########

########## PRE CLONE CREATE ##########

########## UMOUNT COMMANDS ##########

########## Detecting Data OnTap mode for cmode81a-cluster-mgmt ##########[Tue Feb 21 21:34:05 2012] INFO: Data OnTap Cluster mode detected[Tue Feb 21 21:34:05 2012] INFO: Running umount command UMOUNT_CMD01 [AGENT:killall -u mongodb ; exit 0][Tue Feb 21 21:34:06 2012] INFO: Running umount command [killall -u mongodb ; exit 0] finished successfully

########## Detecting Data OnTap mode for cmode81a-cluster-mgmt ##########[Tue Feb 21 21:34:06 2012] INFO: Data OnTap Cluster mode detected[Tue Feb 21 21:34:06 2012] INFO: Running umount command UMOUNT_CMD02 [AGENT:umount /data/mongodb-shard* ; exit 0][Tue Feb 21 21:34:06 2012] INFO: Running umount command [umount /data/mongodb-shard* ; exit 0] finished successfully

########## UMOUNT COMMANDS FINISHED SUCCESSFULLY ##########

########## PRE CLONE COMMANDS ##########[Tue Feb 21 21:34:06 2012] INFO: No commands defined

########## PRE CLONE COMMANDS FINISHED SUCCESSFULLY ##########


########## Creating Volume Clone(s) for source vserver1-mgmt ##########[Tue Feb 21 21:34:06 2012] INFO: Creating Volume Clone cl_clonedb_mongodb_shard1_20120221213400 from snapshot MONGODB-daily_recent of vserver1-mgmt:mongodb_shard1[Tue Feb 21 21:34:12 2012] INFO: Volume Clone Create of cl_clonedb_mongodb_shard1_20120221213400 from MONGODB-daily_recent on vserver1-mgmt Created Successfully[Tue Feb 21 21:34:12 2012] INFO: Creating Volume Clone cl_clonedb_mongodb_shard2_20120221213400 from snapshot MONGODB-daily_recent of vserver1-mgmt:mongodb_shard2[Tue Feb 21 21:34:17 2012] INFO: Volume Clone Create of cl_clonedb_mongodb_shard2_20120221213400 from MONGODB-daily_recent on vserver1-mgmt Created Successfully

########## Gathering Information for luns on vserver1-mgmt ##########

########## Creating export policy sc_clonedb for mongodb-clonedb ##########[Tue Feb 21 21:34:31 2012] INFO: Checking if rule exists for sc_clonedb and mongodb-clonedb on vserver1-mgmt[Tue Feb 21 21:34:31 2012] INFO: NetApp export policy sc_clonedb for mongodb-clonedb already exists

########## Creating export rule for policy sc_clonedb host mongodb-clonedb on vserver1-mgmt ##########[Tue Feb 21 21:34:31 2012] INFO: Checking if rule exists for sc_clonedb and mongodb-clonedb on vserver1-mgmt[Tue Feb 21 21:34:31 2012] INFO: NetApp export policy sc_clonedb host mongodb-clonedb already exists

########## Adding policy sc_clonedb to vserver1-mgmt:cl_clonedb_mongodb_shard1_20120221213400

##########

########## Adding policy sc_clonedb to vserver1-mgmt:cl_clonedb_mongodb_shard2_20120221213400 ##########

########## POST NETAPP DATA TRANSFER COMMANDS ##########[Tue Feb 21 21:34:36 2012] INFO: No commands defined

########## POST NETAPP DATA TRANSFER COMMANDS FINISHED SUCCESSFULLY ##########

########## PRE NETAPP CLONE DELETE COMMANDS ##########[Tue Feb 21 21:34:36 2012] INFO: No commands defined

########## PRE NETAPP CLONE DELETE COMMANDS FINISHED SUCCESSFULLY ##########

########## Removing NetApp Volume Clone(s) for source vserver1-mgmt ##########

########## Running NetApp Snapshot Delete on Primary vserver1-mgmt ##########

########## POST CLONE CREATE ##########

########## POST CREATE COMMANDS ##########[Tue Feb 21 21:35:04 2012] INFO: No commands defined

########## POST CREATE COMMANDS FINISHED SUCCESSFULLY ##########

########## MOUNT COMMANDS ##########

########## Detecting Data OnTap mode for cmode81a-cluster-mgmt ##########[Tue Feb 21 21:35:04 2012] INFO: Data OnTap Cluster mode detected[Tue Feb 21 21:35:04 2012] INFO: Running mount command MOUNT_CMD01 [AGENT:mount -t nfs -o rw,vers=3,rsize=65535,wsize=65535,hard,proto=tcp,timeo=600,retrans=2,sec=sys,addr=172.16.1.101 vserver1-nfs:/cl_clonedb_mongodb_shard1_20120221213400/mongodb_shard1a /data/mongodb-shard1a][Tue Feb 21 21:35:05 2012] INFO: Running mount command [mount -t nfs -o rw,vers=3,rsize=65535,wsize=65535,hard,proto=tcp,timeo=600,retrans=2,sec=sys,addr=172.16.1.101 vserver1-nfs:/cl_clonedb_mongodb_shard1_20120221213400/mongodb_shard1a /data/mongodb-shard1a] finished successfully

########## Detecting Data OnTap mode for cmode81a-cluster-mgmt ##########[Tue Feb 21 21:35:05 2012] INFO: Data OnTap Cluster mode detected[Tue Feb 21 21:35:05 2012] INFO: Running mount command MOUNT_CMD02 [AGENT:mount -t nfs -o rw,vers=3,rsize=65535,wsize=65535,hard,proto=tcp,timeo=600,retrans=2,sec=sys,addr=172.16.1.101 vserver1-nfs:/cl_clonedb_mongodb_shard1_20120221213400/mongodb_shard1b /data/mongodb-shard1b][Tue Feb 21 21:35:05 2012] INFO: Running mount command [mount -t nfs -o rw,vers=3,rsize=65535,wsize=65535,hard,proto=tcp,timeo=600,retrans=2,sec=sys,addr=172.16.1.101 vserver1-nfs:/cl_clonedb_mongodb_shard1_20120221213400/mongodb_shard1b /data/mongodb-shard1b] finished successfully

########## Detecting Data OnTap mode for cmode81a-cluster-mgmt ##########[Tue Feb 21 21:35:05 2012] INFO: Data OnTap Cluster mode detected[Tue Feb 21 21:35:05 2012] INFO: Running mount command MOUNT_CMD03 [AGENT:mount -t nfs -o rw,vers=3,rsize=65535,wsize=65535,hard,proto=tcp,timeo=600,retrans=2,sec=sys,addr=172.16.1.101 vserver1-nfs:/cl_clonedb_mongodb_shard1_20120221213400/mongodb_shard1c /data/mongodb-shard1c][Tue Feb 21 21:35:06 2012] INFO: Running mount command [mount -t nfs -o rw,vers=3,rsize=65535,wsize=65535,hard,proto=tcp,timeo=600,retrans=2,sec=sys,addr=172.16.1.101 vserver1-nfs:/cl_clonedb_mongodb_shard1_20120221213400/mongodb_shard1c /data/mongodb-shard1c] finished successfully

########## Detecting Data OnTap mode for cmode81a-cluster-mgmt ##########[Tue Feb 21 21:35:06 2012] INFO: Data OnTap Cluster mode detected[Tue Feb 21 21:35:06 2012] INFO: Running mount command MOUNT_CMD04 [AGENT:mount -t nfs -o rw,vers=3,rsize=65535,wsize=65535,hard,proto=tcp,timeo=600,retrans=2,sec=sys,addr=172.16.1.101 vserver1-nfs:/cl_clonedb_mongodb_shard2_20120221213400/mongodb_shard2a /data/mongodb-shard2a][Tue Feb 21 21:35:06 2012] INFO: Running mount command [mount -t nfs -o rw,vers=3,rsize=65535,wsize=65535,hard,proto=tcp,timeo=600,retrans=2,sec=sys,addr=172.16.1.101 vserver1-nfs:/cl_clonedb_mongodb_shard2_20120221213400/mongodb_shard2a /data/mongodb-shard2a] finished successfully

########## Detecting Data OnTap mode for cmode81a-cluster-mgmt ##########[Tue Feb 21 21:35:06 2012] INFO: Data OnTap Cluster mode detected[Tue Feb 21 21:35:06 2012] INFO: Running mount command MOUNT_CMD05 [AGENT:mount -t nfs -o rw,vers=3,rsize=65535,wsize=65535,hard,proto=tcp,timeo=600,retrans=2,sec=sys,addr=172.16.1.101 vserver1-nfs:/cl_clonedb_mongodb_shard2_20120221213400/mongodb_shard2b /data/mongodb-shard2b][Tue Feb 21 21:35:07 2012] INFO: Running mount command [mount -t nfs -o rw,vers=3,rsize=65535,wsize=65535,hard,proto=tcp,timeo=600,retrans=2,sec=sys,addr=172.16.1.101 vserver1-nfs:/cl_clonedb_mongodb_shard2_20120221213400/mongodb_shard2b /data/mongodb-shard2b] finished successfully

########## Detecting Data OnTap mode for cmode81a-cluster-mgmt ##########[Tue Feb 21 21:35:07 2012] INFO: Data OnTap Cluster mode detected[Tue Feb 21 21:35:07 2012] INFO: Running mount command MOUNT_CMD06 [AGENT:mount -t nfs -o rw,vers=3,rsize=65535,wsize=65535,hard,proto=tcp,timeo=600,retrans=2,sec=sys,addr=172.16.1.101 vserver1-nfs:/cl_clonedb_mongodb_shard2_20120221213400/mongodb_shard2c /data/mongodb-shard2c][Tue Feb 21 21:35:07 2012] INFO: Running mount command [mount -t nfs -o rw,vers=3,rsize=65535,wsize=65535,hard,proto=tcp,timeo=600,retrans=2,sec=sys,addr=172.16.1.101 vserver1-nfs:/cl_clonedb_mongodb_shard2_20120221213400/mongodb_shard2c /data/mongodb-shard2c] finished successfully

########## Detecting Data OnTap mode for cmode81a-cluster-mgmt ##########[Tue Feb 21 21:35:07 2012] INFO: Data OnTap Cluster mode detected[Tue Feb 21 21:35:07 2012] INFO: Running mount command MOUNT_CMD07 [AGENT:chown -R mongodb:mongodb /data ; exit 0][Tue Feb 21 21:35:08 2012] INFO: Running mount command [chown -R mongodb:mongodb /data ; exit 0] finished successfully

########## Detecting Data OnTap mode for cmode81a-cluster-mgmt ##########[Tue Feb 21 21:35:09 2012] INFO: Data OnTap Cluster mode detected[Tue Feb 21 21:35:09 2012] INFO: Running mount command MOUNT_CMD08 [AGENT:rm -f /data/mongodb-shard*/db/mongod.lock][Tue Feb 21 21:35:09 2012] INFO: Running mount command [rm -f /data/mongodb-shard*/db/mongod.lock] finished successfully

########## Detecting Data OnTap mode for cmode81a-cluster-mgmt ##########[Tue Feb 21 21:35:09 2012] INFO: Data OnTap Cluster mode detected[Tue Feb 21 21:35:09 2012] INFO: Running mount command MOUNT_CMD09 [AGENT:su - mongodb -c '/opt/mongodb/bin/mongod --replSet shard1/mongodb-shard1b:30012,mongodb-shard1c:30013 --journal --rest --shardsvr --port 30011 --dbpath /data/mongodb-shard1a/db --fork --logpath /data/mongodb-shard1a/db/mongod.log'][Tue Feb 21 21:35:10 2012] INFO: Running mount command [su - mongodb -c '/opt/mongodb/bin/mongod --replSet shard1/mongodb-shard1b:30012,mongodb-shard1c:30013 --journal --rest --shardsvr --port 30011 --dbpath /data/mongodb-shard1a/db --fork --logpath /data/mongodb-shard1a/db/mongod.log'] finished successfully

########## Detecting Data OnTap mode for cmode81a-cluster-mgmt ##########[Tue Feb 21 21:35:10 2012] INFO: Data OnTap Cluster mode detected[Tue Feb 21 21:35:10 2012] INFO: Running mount command MOUNT_CMD10 [AGENT:su - mongodb -c '/opt/mongodb/bin/mongod --replSet shard1/mongodb-shard1a:30011,mongodb-shard1c:30013 --journal --rest --shardsvr --port 30012 --dbpath /data/mongodb-shard1b/db --fork --logpath /data/mongodb-shard1b/db/mongod.log'][Tue Feb 21 21:35:11 2012] INFO: Running mount command [su - mongodb -c '/opt/mongodb/bin/mongod --replSet shard1/mongodb-shard1a:30011,mongodb-shard1c:30013 --journal --rest --shardsvr --port 30012 --dbpath /data/mongodb-shard1b/db --fork --logpath /data/mongodb-shard1b/db/mongod.log'] finished successfully

########## Detecting Data OnTap mode for cmode81a-cluster-mgmt ##########[Tue Feb 21 21:35:12 2012] INFO: Data OnTap Cluster mode detected[Tue Feb 21 21:35:12 2012] INFO: Running mount command MOUNT_CMD11 [AGENT:su - mongodb -c '/opt/mongodb/bin/mongod --replSet shard1/mongodb-shard1a:30011,mongodb-shard1b:30012 --journal --rest --shardsvr --port 30013 --dbpath /data/mongodb-shard1c/db --fork --logpath /data/mongodb-shard1c/db/mongod.log'][Tue Feb 21 21:35:13 2012] INFO: Running mount command [su - mongodb -c '/opt/mongodb/bin/mongod --replSet shard1/mongodb-shard1a:30011,mongodb-shard1b:30012 --journal --rest --shardsvr --port 30013 --dbpath /data/mongodb-shard1c/db --fork --logpath /data/mongodb-shard1c/db/mongod.log'] finished successfully

########## Detecting Data OnTap mode for cmode81a-cluster-mgmt ##########[Tue Feb 21 21:35:13 2012] INFO: Data OnTap Cluster mode detected[Tue Feb 21 21:35:13 2012] INFO: Running mount command MOUNT_CMD12 [AGENT:su - mongodb -c '/opt/mongodb/bin/mongod --replSet shard2/mongodb-shard2b:30022,mongodb-shard2c:30023 --journal --rest --shardsvr --port 30021 --dbpath /data/mongodb-shard2a/db --fork --logpath /data/mongodb-shard2a/db/mongod.log'][Tue Feb 21 21:35:14 2012] INFO: Running mount command [su - mongodb -c '/opt/mongodb/bin/mongod --replSet shard2/mongodb-shard2b:30022,mongodb-shard2c:30023 --journal --rest --shardsvr --port 30021 --dbpath /data/mongodb-shard2a/db --fork --logpath /data/mongodb-shard2a/db/mongod.log'] finished successfully

########## Detecting Data OnTap mode for cmode81a-cluster-mgmt ##########[Tue Feb 21 21:35:14 2012] INFO: Data OnTap Cluster mode detected[Tue Feb 21 21:35:14 2012] INFO: Running mount command MOUNT_CMD13 [AGENT:su - mongodb -c '/opt/mongodb/bin/mongod --replSet shard2/mongodb-shard2a:30021,mongodb-shard2c:30023 --journal --rest --shardsvr --port 30022 --dbpath /data/mongodb-shard2b/db --fork --logpath /data/mongodb-shard2b/db/mongod.log'][Tue Feb 21 21:35:17 2012] INFO: Running mount command [su - mongodb -c '/opt/mongodb/bin/mongod --replSet shard2/mongodb-shard2a:30021,mongodb-shard2c:30023 --journal --rest --shardsvr --port 30022 --dbpath /data/mongodb-shard2b/db --fork --logpath /data/mongodb-shard2b/db/mongod.log']

finished successfully

########## Detecting Data OnTap mode for cmode81a-cluster-mgmt ##########[Tue Feb 21 21:35:17 2012] INFO: Data OnTap Cluster mode detected[Tue Feb 21 21:35:17 2012] INFO: Running mount command MOUNT_CMD14 [AGENT:su - mongodb -c '/opt/mongodb/bin/mongod --replSet shard2/mongodb-shard2a:30021,mongodb-shard2b:30022 --journal --rest --shardsvr --port 30023 --dbpath /data/mongodb-shard2c/db --fork --logpath /data/mongodb-shard2c/db/mongod.log'][Tue Feb 21 21:35:19 2012] INFO: Running mount command [su - mongodb -c '/opt/mongodb/bin/mongod --replSet shard2/mongodb-shard2a:30021,mongodb-shard2b:30022 --journal --rest --shardsvr --port 30023 --dbpath /data/mongodb-shard2c/db --fork --logpath /data/mongodb-shard2c/db/mongod.log'] finished successfully

########## Detecting Data OnTap mode for cmode81a-cluster-mgmt ##########[Tue Feb 21 21:35:19 2012] INFO: Data OnTap Cluster mode detected[Tue Feb 21 21:35:19 2012] INFO: Running mount command MOUNT_CMD15 [AGENT:su - mongodb -c '/opt/mongodb/bin/mongod --configsvr --port 4011 --dbpath /data/mongodb-shard1a/configdb/ --fork --logpath /data/mongodb-shard1a/configdb/mongod.log'][Tue Feb 21 21:35:20 2012] INFO: Running mount command [su - mongodb -c '/opt/mongodb/bin/mongod --configsvr --port 4011 --dbpath /data/mongodb-shard1a/configdb/ --fork --logpath /data/mongodb-shard1a/configdb/mongod.log'] finished successfully

########## Detecting Data OnTap mode for cmode81a-cluster-mgmt ##########[Tue Feb 21 21:35:20 2012] INFO: Data OnTap Cluster mode detected[Tue Feb 21 21:35:20 2012] INFO: Running mount command MOUNT_CMD16 [AGENT:su - mongodb -c '/opt/mongodb/bin/mongod --configsvr --port 4021 --dbpath /data/mongodb-shard2a/configdb/ --fork --logpath /data/mongodb-shard2a/configdb/mongod.log'][Tue Feb 21 21:35:22 2012] INFO: Running mount command [su - mongodb -c '/opt/mongodb/bin/mongod --configsvr --port 4021 --dbpath /data/mongodb-shard2a/configdb/ --fork --logpath /data/mongodb-shard2a/configdb/mongod.log'] finished successfully

########## Detecting Data OnTap mode for cmode81a-cluster-mgmt ##########[Tue Feb 21 21:35:22 2012] INFO: Data OnTap Cluster mode detected[Tue Feb 21 21:35:22 2012] INFO: Running mount command MOUNT_CMD17 [AGENT:su - mongodb -c '/opt/mongodb/bin/mongod --configsvr --port 4023 --dbpath /data/mongodb-shard2c/configdb/ --fork --logpath /data/mongodb-shard2c/configdb/mongod.log'][Tue Feb 21 21:35:23 2012] INFO: Running mount command [su - mongodb -c '/opt/mongodb/bin/mongod --configsvr --port 4023 --dbpath /data/mongodb-shard2c/configdb/ --fork --logpath /data/mongodb-shard2c/configdb/mongod.log'] finished successfully

########## Detecting Data OnTap mode for cmode81a-cluster-mgmt ##########[Tue Feb 21 21:35:23 2012] INFO: Data OnTap Cluster mode detected[Tue Feb 21 21:35:23 2012] INFO: Running mount command MOUNT_CMD18 [AGENT:su - mongodb -c '/opt/mongodb/bin/mongod --configsvr --port 4023 --dbpath /data/mongodb-shard2c/configdb/ --fork --logpath /data/mongodb-shard2c/configdb/mongod.log'][Tue Feb 21 21:35:24 2012] INFO: Running mount command [su - mongodb -c '/opt/mongodb/bin/mongod --configsvr --port 4023 --dbpath /data/mongodb-shard2c/configdb/ --fork --logpath /data/mongodb-shard2c/configdb/mongod.log'] finished successfully

########## Detecting Data OnTap mode for cmode81a-cluster-mgmt ##########[Tue Feb 21 21:35:24 2012] INFO: Data OnTap Cluster mode detected[Tue Feb 21 21:35:24 2012] INFO: Running mount command MOUNT_CMD19 [AGENT:su - mongodb -c 'sleep 60; /opt/mongodb/bin/mongos --configdb mongodb-clonedb:4011,mongodb-clonedb:4021,mongodb-clonedb:4023 --fork --logpath /data/mongos.log'][Tue Feb 21 21:36:24 2012] INFO: Running mount command [su - mongodb -c 'sleep 60; /opt/mongodb/bin/mongos --configdb mongodb-clonedb:4011,mongodb-clonedb:4021,mongodb-clonedb:4023 --fork --logpath /data/mongos.log'] finished successfully

########## MOUNT COMMANDS FINISHED SUCCESSFULLY ##########

########## APPLICATION CLONE FOLLOW UP COMMANDS ##########[Tue Feb 21 21:36:24 2012] INFO: No commands defined

########## APPLICATION CLONE FOLLOW UP COMMANDS FINISHED SUCCESSFULLY ##########

########## POST NETAPP COMMANDS ##########[Tue Feb 21 21:36:24 2012] INFO: No commands defined

########## POST NETAPP COMMANDS FINISHED SUCCESSFULLY ##########

########## ARCHIVE COMMANDS ##########[Tue Feb 21 21:36:24 2012] INFO: No commands defined

########## ARCHIVE COMMANDS FINISHED SUCCESSFULLY ##########

########## NetApp Snap Creator Framework 3.5.0 finished successfully ##########[Tue Feb 21 21:36:24 2012] INFO: INFO: Snap Creator finished successfully ( Action: clone_vol )

Review cloned database

We now have a working cloned copy of the production MongoDB database, isolated and available for writes, troubleshooting or dev-test, and since its storage is based on flexclone technology, no additional storage was used in the process.

Running 'df' on the clonedb shows that all the replicaset qtrees were mounted on a modified mountpoint structure..

[root@mongodb-clonedb ~]# df Filesystem 1K-blocks Used Available Use% Mounted on /dev/simfs 2097152 859308 1237844 41% / none 262144 4 262140 1% /dev vserver1-nfs:/cl_clonedb_mongodb_shard1_20120221213400/mongodb_shard1a 2988448 505472 2482976 17% /data/mongodb-shard1a vserver1-nfs:/cl_clonedb_mongodb_shard1_20120221213400/mongodb_shard1b 2988448 505472 2482976 17% /data/mongodb-shard1b vserver1-nfs:/cl_clonedb_mongodb_shard1_20120221213400/mongodb_shard1c 2988448 505472 2482976 17% /data/mongodb-shard1c vserver1-nfs:/cl_clonedb_mongodb_shard2_20120221213400/mongodb_shard2a 2988448 275680 2712768 10% /data/mongodb-shard2a vserver1-nfs:/cl_clonedb_mongodb_shard2_20120221213400/mongodb_shard2b 2988448 275680 2712768 10% /data/mongodb-shard2b vserver1-nfs:/cl_clonedb_mongodb_shard2_20120221213400/mongodb_shard2c 2988448 275680 2712768 10% /data/mongodb-shard2c [root@mongodb-clonedb ~]#

All processes were automatically started, together with a mongos process running locally. This mongos process can be used to query or update the database clone. Or even for performing restores into production as we'll see in the next example.

[root@mongodb-clonedb ~]# ps -ef | grep mongo mongodb 5104 1 8 00:35 ? 00:01:43 /opt/mongodb/bin/mongod --replSet shard1/mongodb-shard1b:30012,mongodb-shard1c:30013 --journal --rest --shardsvr --port 30011 --dbpath /data/mongodb-shard1a/db --fork --logpath /data/mongodb-shard1a/db/mongod.log mongodb 5132 1 0 00:35 ? 00:00:07 /opt/mongodb/bin/mongod --replSet shard1/mongodb-shard1a:30011,mongodb-shard1c:30013 --journal --rest --shardsvr --port 30012 --dbpath /data/mongodb-shard1b/db --fork --logpath /data/mongodb-shard1b/db/mongod.log mongodb 5162 1 0 00:35 ? 00:00:07 /opt/mongodb/bin/mongod --replSet shard1/mongodb-shard1a:30011,mongodb-shard1b:30012 --journal --rest --shardsvr --port 30013 --dbpath /data/mongodb-shard1c/db --fork --logpath /data/mongodb-shard1c/db/mongod.log mongodb 5202 1 8 00:35 ? 00:01:43 /opt/mongodb/bin/mongod --replSet shard2/mongodb-shard2b:30022,mongodb-shard2c:30023 --journal --rest --shardsvr --port 30021 --dbpath /data/mongodb-shard2a/db --fork --logpath /data/mongodb-shard2a/db/mongod.log mongodb 5255 1 0 00:35 ? 00:00:07 /opt/mongodb/bin/mongod --replSet shard2/mongodb-shard2a:30021,mongodb-shard2c:30023 --journal --rest --shardsvr --port 30022 --dbpath /data/mongodb-shard2b/db --fork --logpath /data/mongodb-shard2b/db/mongod.log mongodb 5293 1 0 00:35 ? 00:00:06 /opt/mongodb/bin/mongod --replSet shard2/mongodb-shard2a:30021,mongodb-shard2b:30022 --journal --rest --shardsvr --port 30023 --dbpath /data/mongodb-shard2c/db --fork --logpath /data/mongodb-shard2c/db/mongod.log mongodb 5336 1 0 00:35 ? 00:00:06 /opt/mongodb/bin/mongod --configsvr --port 4011 --dbpath /data/mongodb-shard1a/configdb/ --fork --logpath /data/mongodb-shard1a/configdb/mongod.log mongodb 5381 1 0 00:35 ? 00:00:06 /opt/mongodb/bin/mongod --configsvr --port 4021 --dbpath /data/mongodb-shard2a/configdb/ --fork --logpath /data/mongodb-shard2a/configdb/mongod.log mongodb 5406 1 0 00:35 ? 00:00:06 /opt/mongodb/bin/mongod --configsvr --port 4023 --dbpath /data/mongodb-shard2c/configdb/ --fork --logpath /data/mongodb-shard2c/configdb/mongod.log mongodb 5527 1 0 00:36 ? 00:00:04 /opt/mongodb/bin/mongos --configdb mongodb-clonedb:4011,mongodb-clonedb:4021,mongodb-clonedb:4023 --fork --logpath /data/mongos.log root 6469 601 0 00:54 pts/0 00:00:00 grep mongo [root@mongodb-clonedb ~]#

SECTION:PARTIAL-RESTORE: RESTORING A SET OF DOCUMENTS

By using our new flexcloned database, its fairly easy to restore a set of documents on the production database. The process is described here is a simple case of reading the data from the clone and importing it back into production.

In this example, we'll simulate some corruption or loss of data by deleting all documents related to week 42 in the production database.

Our restore process is based on a python script which loops through all week 42's documents in the cloned database and inserts them back in the production database.

These types of restores (imports from clones) can be fairly quick since writes in MongoDB are quite fast and we can leverage reads across all members of a replicaset.

Connect to the production database[root@appserver scServer3.5.0]# su - mongodb [mongodb@appserver ~]$ /opt/mongodb/bin/mongo appserver:27017 MongoDB shell version: 2.0.2 connecting to: appserver:27017/test mongos> use lottery switched to db lottery mongos>

Check number of documents associated with week 42mongos> db.tickets.count({week:42}) 10000

Remove all objects from that weekmongos> db.tickets.remove({"week":42}) mongos>

Double-checkmongos> db.tickets.count({week:42}) 0 mongos>

Copy the restore python script:

restore-objects.pyimport random import time import sys from pymongo import Connection

week=int(sys.argv[1]) src_ip=sys.argv[2] src_p=int(sys.argv[3]) dst_ip=sys.argv[4] dst_p=int(sys.argv[5])

#connection = Connection('appserver', 27017 ) src_connection = Connection(src_ip, src_p ) dst_connection = Connection(dst_ip, dst_p )

db_src = src_connection.lottery #connect to source db_dst = dst_connection.lottery #connect to destination

for entry in db_src.tickets.find({"week": week}) : print entry db_dst.tickets.save(entry)

Execute the restorepython2 restore-objects.py 42 mongodb-clonedb 27017 appserver 27017...'ticketnumber': 531590, u'_id': ObjectId('4f3b0a79466d6f30780045d3')} {u'lnhash': 15816876075L, u'week': 42, u'agency': 539, u'luckynumbers': [39, 17, 25, 21, 27, 11, 9, 17], u'ticketnumber': 842205, u'_id': ObjectId('4f3b0a79466d6f3078004640')} {u'lnhash': 16177033808L, u'week': 42, u'agency': 589, u'luckynumbers': [1, 19, 38, 26, 46, 29, 34, 19], u'ticketnumber': 518517, u'_id': ObjectId('4f3b0a79466d6f307800464e')} {u'lnhash': 16159248000L, u'week': 42, u'agency': 251, u'luckynumbers': [41, 35, 32, 23, 10, 3, 30, 17], u'ticketnumber': 489334, u'_id': ObjectId('4f3b0a7b466d6f30780048de')} {u'lnhash': 16284457728L, u'week': 42, u'agency': 785, u'luckynumbers': [47, 4, 9, 31, 21, 14, 32, 33], u'ticketnumber': 725647, u'_id': ObjectId('4f3b0a7b466d6f3078004990')} {u'lnhash': 15817052160L, u'week': 42, u'agency': 782, u'luckynumbers': [16, 2, 24, 35, 24, 41, 23, 26], u'ticketnumber': 231903, u'_id': ObjectId('4f3b0a7c466d6f3078004adf')} {u'lnhash': 16001356800L, u'week': 42, u'agency': 686, u'luckynumbers': [31, 39, 1, 22, 25, 47, 16, 32], u'ticketnumber': 674687, u'_id': ObjectId('4f3b0a7c466d6f3078004b57')} {u'lnhash': 15944040000L, u'week': 42, u'agency': 326, u'luckynumbers': [20, 35, 27, 1, 15, 38, 40, 37], u'ticketnumber': 959939, u'_id': ObjectId('4f3b0a7c466d6f3078004c04')} {u'lnhash': 15840126720L, u'week': 42, u'agency': 19, u'luckynumbers': [36, 43, 33, 10, 8, 6, 17, 38], u'ticketnumber': 548556, u'_id': ObjectId('4f3b0a7d466d6f3078004c2b')} {u'lnhash': 16231767552L, u'week': 42, u'agency': 113, u'luckynumbers': [11, 17, 24, 32, 21, 39, 46, 3], u'ticketnumber': 846127, u'_id': ObjectId('4f3b0a7d466d6f3078004cfb')}

Connect to production database[mongodb@appserver ~]$ /opt/mongodb/bin/mongo appserver:27017 MongoDB shell version: 2.0.2 connecting to: appserver:27017/test mongos> use lottery switched to db lottery

Check for the restored objectsmongos> db.tickets.count({week:42}) 10000 mongos>

SECTION: FULL RESTORE EXAMPLE

As the size of the database increases, problems in backing up and restoring it arise. SnapShot technology such as NetApp Snapshots is the most simple and efficient way of backing up and restoring a distributed database of any size.

As backups are based on snapshots, the time to restore a large database composed of multiple shards is directly related to the time spent in stopping and starting the database processes than the act of restoring the data itself. The restore of a snapshot only takes a fraction of a second.

For this example, we'll remove our tickets collection, stop all mongodb processes across all the shards, umount the /data filesystems, restore the volumes using snapcreator, remount the filesystems, remove lock files and restart all processes.

Connect to production, and drop the tickets collection[mongodb@appserver ~]$ /opt/mongodb/bin/mongo appserver:27017 MongoDB shell version: 2.0.2 connecting to: appserver:27017/test mongos> use lottery switched to db lottery mongos> db.tickets.count({week:42}) 10000 mongos> db.tickets.drop() true mongos>

mongos> db.tickets.count() 0 mongos>

Kill all mongodb processes

Login to every node and execute:[root@mongodb-shard1a /]# kill -9 -p $( ps -ef | grep /opt/mongodb/bin | grep -v grep | awk '{ print $2 }')

[root@mongodb-shard1b /]# kill -9 -p $( ps -ef | grep /opt/mongodb/bin | grep -v grep | awk '{ print $2 }')

[root@mongodb-shard1c /]# kill -9 -p $( ps -ef | grep /opt/mongodb/bin | grep -v grep | awk '{ print $2 }')

[root@mongodb-shard2a /]# kill -9 -p $( ps -ef | grep /opt/mongodb/bin | grep -v grep | awk '{ print $2 }')

[root@mongodb-shard2b /]# kill -9 -p $( ps -ef | grep /opt/mongodb/bin | grep -v grep | awk '{ print $2 }')

[root@mongodb-shard2c /]# kill -9 -p $( ps -ef | grep /opt/mongodb/bin | grep -v grep | awk '{ print $2 }')

Umount filesystems on every node[root@mongodb-shard1a /]# umount /data [root@mongodb-shard1a /]#

[root@mongodb-shard1b /]# umount /data [root@mongodb-shard1b /]#

[root@mongodb-shard1c /]# umount /data [root@mongodb-shard1c /]#

[root@mongodb-shard2 /]# umount /data [root@mongodb-shard2 /]#

[root@mongodb-shard2b /]# umount /data

[root@mongodb-shard2b /]#

[root@mongodb-shard2c /]# umount /data [root@mongodb-shard2c /]#

Retrieve the list of available snapshots[root@appserver scServer3.5.0]# /usr/local/scServer3.5.0/snapcreator --profile mongodb --config master --action snaplist





Restore the data from the snapshots using SnapCreatorroot@appserver scServer3.5.0]# /usr/local/scServer3.5.0/snapcreator --profile mongodb --config master --action restore --policy daily --snap_name MONGODB-daily_recent --verbose

[Wed Feb 22 00:50:41 2012] INFO: Logfile timestamp: 20120222005041


########## PRE RESTORE COMMANDS ########## [Wed Feb 22 00:50:41 2012] INFO: No commands defined

########## PRE RESTORE COMMANDS FINISHED SUCCESSFULLY ##########

### You have chosen to do a snap restore on one or more volumes for the Config: master Policy: daily ### Are you sure you want to continue (y|n)? y########## Detecting Data OnTap mode for vserver1-mgmt ########## [Wed Feb 22 00:53:39 2012] INFO: Data OnTap Cluster mode detected

########## Detecting Data OnTap mode for vserver1-mgmt ########## [Wed Feb 22 00:53:39 2012] INFO: Data OnTap Cluster mode detected

### Volume Menu for vserver1-mgmt ### 01. mongodb_shard1 02. mongodb_shard2

Select a volume for snapshot restore (enter a number, "n" for next filer, "c" to continue, or "q" to quit): 01########## Gathering Information for vserver1-mgmt:mongodb_shard1 ########## [Wed Feb 22 00:53:45 2012] INFO: Performing NetApp Snapshot Inventory for mongodb_shard1 on vserver1-mgmt [Wed Feb 22 00:53:46 2012] INFO: NetApp Snapshot Inventory of mongodb_shard1 on vserver1-mgmt completed Successfully

### Snapshot Menu for vserver1-mgmt:mongodb_shard1 ### 01. MONGODB-daily_recent (Mar 01 2012 14:48:04) 02. MONGODB-daily_20120301135116 (Mar 01 2012 13:51:16)

03. MONGODB-daily_20120229140023 (Feb 29 2012 14:00:23) 04. MONGODB-daily_20120229135802 (Feb 29 2012 13:58:02) 05. MONGODB-daily_20120228163417 (Feb 28 2012 16:34:17) 06. MONGODB-daily_20120228152757 (Feb 28 2012 15:27:57) 07. MONGODB-daily_20120228152346 (Feb 28 2012 15:23:46) 08. MONGODB-daily_20120228135612 (Feb 28 2012 13:56:12) 09. MONGODB-daily_20120228134412 (Feb 28 2012 13:44:12) Select a snapshot for restore (enter a number or "q" to quit): 01 ### Restore Menu for vserver1-mgmt:mongodb_shard1 snapshot MONGODB-daily_recent ### 01. Volume Restore 02. Single File Restore Select a restore type (enter a number, or "q" to quit): 01 WARN: You have selected to do a volume restore, All data in vserver1-mgmt:mongodb_shard1 will be reverted to snapshot MONGODB-daily_recent WARN: All data in vserver1-mgmt:mongodb_shard1 from Mar 01 2012 14:48:04 to Feb 22 2012 00:54:04 will be lost!!! Are you sure you want to continue with the restore (y|n)? y[Wed Feb 22 00:54:35 2012] INFO: NetApp Snapshot Volume Restore of MONGODB-daily_recent on vserver1-mgmt:mongodb_shard1 finished successfully

### Volume Menu for vserver1-mgmt ### 01. mongodb_shard1 02. mongodb_shard2

Select a volume for snapshot restore (enter a number, "n" for next filer, "c" to continue, or "q" to quit): 02 ########## Gathering Information for vserver1-mgmt:mongodb_shard2 ########## [Wed Feb 22 00:54:55 2012] INFO: Performing NetApp Snapshot Inventory for mongodb_shard2 on vserver1-mgmt [Wed Feb 22 00:54:55 2012] INFO: NetApp Snapshot Inventory of mongodb_shard2 on vserver1-mgmt completed Successfully

### Snapshot Menu for vserver1-mgmt:mongodb_shard2 ### 01. MONGODB-daily_recent (Mar 01 2012 14:48:06) (busy,vclone) 02. MONGODB-daily_20120301135117 (Mar 01 2012 13:51:17) 03. MONGODB-daily_20120229140026 (Feb 29 2012 14:00:26) 04. MONGODB-daily_20120229135804 (Feb 29 2012 13:58:04) 05. MONGODB-daily_20120228163419 (Feb 28 2012 16:34:19) 06. MONGODB-daily_20120228152758 (Feb 28 2012 15:27:58) 07. MONGODB-daily_20120228152348 (Feb 28 2012 15:23:48) 08. MONGODB-daily_20120228135614 (Feb 28 2012 13:56:14) 09. MONGODB-daily_20120228134414 (Feb 28 2012 13:44:14) Select a snapshot for restore (enter a number or "q" to quit): 01 ### Restore Menu for vserver1-mgmt:mongodb_shard2 snapshot MONGODB-daily_recent ### 01. Volume Restore 02. Single File Restore Select a restore type (enter a number, or "q" to quit): 01 WARN: You have selected to do a volume restore, All data in vserver1-mgmt:mongodb_shard2 will be reverted to snapshot MONGODB-daily_recent WARN: All data in vserver1-mgmt:mongodb_shard2 from Mar 01 2012 14:48:06 to Feb 22 2012 00:55:06 will be lost!!! Are you sure you want to continue with the restore (y|n)?y [Wed Feb 22 00:55:34 2012] INFO: NetApp Snapshot Volume Restore of MONGODB-daily_recent on vserver1-mgmt:mongodb_shard2 finished successfully

### Volume Menu for vserver1-mgmt ### 01. mongodb_shard1 02. mongodb_shard2 Select a volume for snapshot restore (enter a number, "n" for next filer, "c" to continue, or "q" to quit): q

Start the mongod processes across all servers

mongod-shard1a

Mount the filesystem /data and remove the stale lockfiles[root@mongodb-shard1a /]# mount /data [root@mongodb-shard1a /]# su – mongodb -bash-4.1$ rm /data/db/mongod.lock -bash-4.1$ rm /data/configdb/mongod.lock

Restart the mongodb processes-bash-4.1$ /opt/mongodb/bin/mongod --replSet shard1/mongodb-shard1b:30012,mongodb-shard1c:30013

--journal --rest --shardsvr –-port 30011 --fork --logpath /data/db/mongod.log -bash-4.1$ forked process: 16668 all output going to: /data/db/mongod.log

Check the logfiles for errors on startup-bash-4.1$ tail /data/db/mongod.log Wed Feb 22 04:12:12 [initandlisten] Wed Feb 22 04:12:12 [initandlisten] db version v2.0.2, pdfile version 4.5 Wed Feb 22 04:12:12 [initandlisten] git version: 514b122d308928517f5841888ceaa4246a7f18e3 Wed Feb 22 04:12:12 [initandlisten] build info: Linux bs-linux64.10gen.cc 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_41 Wed Feb 22 04:12:12 [initandlisten] options: { fork: true, journal: true, logpath: "/data/db/mongod.log", port: 30011, replSet: "shard1/mongodb-shard1b:30012,mongodb-shard1c:30013", rest: true, shardsvr: true } Wed Feb 22 04:12:12 [initandlisten] journal dir=/data/db/journal Wed Feb 22 04:12:12 [initandlisten] recover : no journal files present, no recovery needed Wed Feb 22 04:12:16 [initandlisten] waiting for connections on port 30011 Wed Feb 22 04:12:16 [websvr] admin web console waiting for connections on port 31011 Wed Feb 22 04:12:16 [initandlisten] connection accepted from 127.0.0.1:48874 #1

Start the arbiter process -bash-4.1$ /opt/mongodb/bin/mongod --configsvr --port 27019 --fork --logpath /data/configdb/arbiter.log -bash-4.1$ forked process: 17034 all output going to: /data/configdb/arbiter.log

Check the logfiles for errors on startup-bash-4.1$ tail /data/configdb/arbiter.log Wed Feb 22 04:26:46 [initandlisten] recover skipping application of section seq:148335 < lsn:1670482 Wed Feb 22 04:26:46 [initandlisten] recover skipping application of section seq:148425 < lsn:1670482 Wed Feb 22 04:26:46 [initandlisten] recover skipping application of section seq:175934 < lsn:1670482 Wed Feb 22 04:26:46 [initandlisten] recover skipping application of section seq:176104 < lsn:1670482 Wed Feb 22 04:26:46 [initandlisten] recover skipping application of section more... Wed Feb 22 04:26:46 [initandlisten] recover cleaning up Wed Feb 22 04:26:46 [initandlisten] removeJournalFiles Wed Feb 22 04:26:46 [initandlisten] recover done Wed Feb 22 04:26:47 [initandlisten] waiting for connections on port 27019 Wed Feb 22 04:26:47 [websvr] admin web console waiting for connections on port 28019 -bash-4.1$

mongod-shard1b

Mount the filesystem /data and remove the stale lockfiles[root@mongodb-shard1b /]# mount /data [root@mongodb-shard1b /]# su – mongodb -bash-4.1$ rm /data/db/mongod.lock

Restart the mongodb processes-bash-4.1$ /opt/mongodb/bin/mongod --replSet shard1/mongodb-shard1a:30011,mongodb-shard1c:30012 --journal --rest --shardsvr –-port 30012 --fork --logpath /data/db/mongod.log -bash-4.1$ forked process: 15966 all output going to: /data/db/mongod.log

Check the logfiles for errors on startup -bash-4.1$ tail /data/db/mongod.log Wed Feb 22 04:15:03 [rsHealthPoll] replSet member mongodb-shard1a:30011 is up Wed Feb 22 04:15:03 [rsHealthPoll] replSet member mongodb-shard1a:30011 is now in state SECONDARY Wed Feb 22 04:15:03 [rsHealthPoll] couldn't connect to mongodb-shard1c:30013: couldn't connect to server mongodb-shard1c:30013 Wed Feb 22 04:15:03 [rsHealthPoll] replSet info mongodb-shard1c:30013 is down (or slow to respond): socket exception Wed Feb 22 04:15:03 [rsHealthPoll] replSet member mongodb-shard1c:30013 is now in state DOWN Wed Feb 22 04:15:03 [rsMgr] not electing self, mongodb-shard1a:30011 would veto Wed Feb 22 04:15:03 [rsMgr] not electing self, mongodb-shard1a:30011 would veto Wed Feb 22 04:15:08 [conn2] replSet info voting yea for mongodb-shard1a:30011 (0) Wed Feb 22 04:15:09 [rsHealthPoll] replSet member mongodb-shard1a:30011 is now in state PRIMARY Wed Feb 22 04:15:13 [rsSync] replSet syncing to: mongodb-shard1a:30011 -bash-4.1$

mongod-shard1c

Mount the filesystem /data and remove the stale lockfiles[root@mongodb-shard1c /]# mount /data [root@mongodb-shard1c /]# su - mongodb -bash-4.1$ rm /data/db/mongod.lock -bash-4.1$

Restart the mongodb processes

-bash-4.1$ /opt/mongodb/bin/mongod --replSet shard1/mongodb-shard1b:30012,mongodb-shard1a:30011 --journal --rest --shardsvr --port 30013 --fork --logpath /data/db/mongod.log -bash-4.1$ forked process: 15707 all output going to: /data/db/mongod.log

-bash-4.1$

Check the logfiles for errors on startup -bash-4.1$ tail /data/db/mongod.log Wed Feb 22 04:16:40 [rsHealthPoll] replSet member mongodb-shard1a:30011 is now in state PRIMARY Wed Feb 22 04:16:40 [rsStart] replSet STARTUP2 Wed Feb 22 04:16:40 [initandlisten] connection accepted from 172.16.1.91:53707 #2 Wed Feb 22 04:16:41 [rsSync] replSet SECONDARY Wed Feb 22 04:16:41 [initandlisten] connection accepted from 172.16.1.92:47514 #3 Wed Feb 22 04:16:45 [rsSync] replSet syncing to: mongodb-shard1a:30011 Wed Feb 22 04:16:46 [conn2] end connection 172.16.1.91:53707 Wed Feb 22 04:16:46 [initandlisten] connection accepted from 172.16.1.91:53718 #4 Wed Feb 22 04:17:01 [conn3] end connection 172.16.1.92:47514 Wed Feb 22 04:17:01 [initandlisten] connection accepted from 172.16.1.92:47527 #5 -bash-4.1$

mongod-shard2a

Mount the filesystem /data and remove the stale lockfiles[root@mongodb-shard2a /]# mount /data [root@mongodb-shard2a /]# su - mongodb -bash-4.1$ rm /data/db/mongod.lock -bash-4.1$ rm /data/configdb/mongod.lock -bash-4.1$

Restart the mongodb processes-bash-4.1$ /opt/mongodb/bin/mongod --replSet shard2/mongodb-shard2b:30022,mongodb-shard2c:30023 --journal --rest --shardsvr --port 30021 --fork --logpath /data/db/mongod.log -bash-4.1$ forked process: 16457 all output going to: /data/db/mongod.log

Check the logfiles for errors on startup -bash-4.1$ tail /data/db/mongod.log Wed Feb 22 04:19:21 [rsHealthPoll] replSet member mongodb-shard2c:30023 is now in state DOWN Wed Feb 22 04:19:21 [rsHealthPoll] replSet info mongodb-shard2b:30022 is down (or slow to respond): socket exception Wed Feb 22 04:19:21 [rsHealthPoll] replSet member mongodb-shard2b:30022 is now in state DOWN Wed Feb 22 04:19:21 [rsStart] replSet STARTUP2 Wed Feb 22 04:19:22 [rsSync] replSet SECONDARY Wed Feb 22 04:19:27 [rsMgr] replSet can't see a majority, will not try to elect self Wed Feb 22 04:19:37 [rsSync] waiting for 6 pings from other members before syncing Wed Feb 22 04:19:49 [rsHealthPoll] couldn't connect to mongodb-shard2c:30023: couldn't connect to server mongodb-shard2c:30023 Wed Feb 22 04:19:49 [rsHealthPoll] couldn't connect to mongodb-shard2b:30022: couldn't connect to server mongodb-shard2b:30022 Wed Feb 22 04:19:52 [rsSync] waiting for 6 pings from other members before syncing -bash-4.1$

Start the arbiter process -bash-4.1$ /opt/mongodb/bin/mongod --configsvr --port 27019 --fork --logpath /data/configdb/arbiter.log -bash-4.1$ forked process: 16703

all output going to: /data/configdb/arbiter.log

Check the logfiles for errors on startup

-bash-4.1$ tail /data/configdb/arbiter.log Wed Feb 22 04:27:37 [initandlisten] recover skipping application of section seq:2214848 < lsn:8825523 Wed Feb 22 04:27:37 [initandlisten] recover skipping application of section seq:2341718 < lsn:8825523 Wed Feb 22 04:27:37 [initandlisten] recover skipping application of section seq:2381157 < lsn:8825523 Wed Feb 22 04:27:37 [initandlisten] recover skipping application of section seq:2425077 < lsn:8825523 Wed Feb 22 04:27:37 [initandlisten] recover skipping application of section more... Wed Feb 22 04:27:39 [initandlisten] recover cleaning up Wed Feb 22 04:27:39 [initandlisten] removeJournalFiles Wed Feb 22 04:27:39 [initandlisten] recover done Wed Feb 22 04:27:40 [initandlisten] waiting for connections on port 27019 Wed Feb 22 04:27:40 [websvr] admin web console waiting for connections on port 28019

mongod-shard2b

Mount the filesystem /data and remove the stale lockfiles[root@mongodb-shard2b /]# mount /data [root@mongodb-shard2b /]# su - mongodb -bash-4.1$ rm /data/db/mongod.lock -bash-4.1$

Restart the mongodb processes-bash-4.1$ /opt/mongodb/bin/mongod --replSet shard2/mongodb-shard2a:30021,mongodb-shard2c:30023 --journal --rest --shardsvr --port 30022 --fork --logpath /data/db/mongod.log -bash-4.1$ forked process: 15712 all output going to: /data/db/mongod.log

Check the logfiles for errors on startup -bash-4.1$ tail /data/db/mongod.log Wed Feb 22 04:21:04 [rsStart] couldn't connect to mongodb-shard2c:30023: couldn't connect to server mongodb-shard2c:30023 Wed Feb 22 04:21:04 [rsHealthPoll] replSet info mongodb-shard2c:30023 is down (or slow to respond): socket exception Wed Feb 22 04:21:04 [rsHealthPoll] replSet member mongodb-shard2c:30023 is now in state DOWN Wed Feb 22 04:21:04 [rsHealthPoll] replSet member mongodb-shard2a:30021 is up Wed Feb 22 04:21:04 [rsHealthPoll] replSet member mongodb-shard2a:30021 is now in state SECONDARY Wed Feb 22 04:21:04 [rsStart] replSet STARTUP2 Wed Feb 22 04:21:05 [rsSync] replSet SECONDARY Wed Feb 22 04:21:05 [initandlisten] connection accepted from 172.16.1.94:50200 #2 Wed Feb 22 04:21:08 [conn2] replSet info voting yea for mongodb-shard2a:30021 (0) Wed Feb 22 04:21:08 [rsHealthPoll] replSet member mongodb-shard2a:30021 is now in state PRIMARY -bash-4.1$

mongod-shard2c

Mount the filesystem /data and remove the stale lockfiles

[root@mongodb-shard2c /]# mount /data [root@mongodb-shard2c /]# su - mongodb -bash-4.1$ rm /data/db/mongod.lock -bash-4.1$ rm /data/configdb/mongod.lock -bash-4.1$

Restart the mongodb processes

-bash-4.1$ /opt/mongodb/bin/mongod --replSet shard2/mongodb-shard1b:30022,mongodb-shard2a:30021 --journal --rest --shardsvr --port 30023 --fork --logpath /data/db/mongod.log -bash-4.1$ forked process: 15928 all output going to: /data/db/mongod.log

Check the logfiles for errors on startup -bash-4.1$ tail /data/db/mongod.log

Wed Feb 22 04:22:27 [rsHealthPoll] replSet member mongodb-shard2a:30021 is now in state PRIMARY Wed Feb 22 04:22:27 [rsHealthPoll] replSet member mongodb-shard2b:30022 is up Wed Feb 22 04:22:27 [rsHealthPoll] replSet member mongodb-shard2b:30022 is now in state SECONDARY Wed Feb 22 04:22:27 [rsStart] replSet STARTUP2 Wed Feb 22 04:22:27 [rsSync] replSet SECONDARY Wed Feb 22 04:22:27 [initandlisten] connection accepted from 172.16.1.94:58913 #2 Wed Feb 22 04:22:28 [initandlisten] connection accepted from 172.16.1.95:48017 #3 Wed Feb 22 04:22:31 [rsSync] replSet syncing to: mongodb-shard2a:30021 Wed Feb 22 04:22:32 [conn3] end connection 172.16.1.95:48017 Wed Feb 22 04:22:32 [initandlisten] connection accepted from 172.16.1.95:48022 #4

Start the arbiter process -bash-4.1$ /opt/mongodb/bin/mongod --configsvr --port 27019 --fork --logpath /data/configdb/arbiter.log -bash-4.1$ forked process: 16018 all output going to: /data/configdb/arbiter.log

Check the logfiles for errors on startup -bash-4.1$ tail /data/configdb/arbiter.log Wed Feb 22 04:28:43 [initandlisten] recover skipping application of section seq:591135 < lsn:6430830 Wed Feb 22 04:28:43 [initandlisten] recover skipping application of section seq:642415 < lsn:6430830 Wed Feb 22 04:28:43 [initandlisten] recover skipping application of section seq:669425 < lsn:6430830 Wed Feb 22 04:28:43 [initandlisten] recover skipping application of section seq:689466 < lsn:6430830 Wed Feb 22 04:28:43 [initandlisten] recover skipping application of section more... Wed Feb 22 04:28:47 [initandlisten] recover cleaning up Wed Feb 22 04:28:47 [initandlisten] removeJournalFiles Wed Feb 22 04:28:47 [initandlisten] recover done Wed Feb 22 04:28:48 [initandlisten] waiting for connections on port 27019 Wed Feb 22 04:28:48 [websvr] admin web console waiting for connections on port 28019

Start the Mongos process [mongodb@appserver ~]$ /opt/mongodb/bin/mongos --configdb mongodb-config1:27019,mongodb-config2:27019,mongodb-config3:27019 --logpath /data/mongos.log --fork --port 27017 [mongodb@appserver ~]$ forked process: 137197 all output going to: /data/mongos.log

Check the logfiles for errors on startup [mongodb@appserver ~]$ tail /data/mongos.log Wed Feb 22 01:30:37 [Balancer] updated set (shard1) to: shard1/mongodb-shard1c:30013,mongodb-shard1c:30013,mongodb-shard1b:30012,mongodb-shard1a:30011 Wed Feb 22 01:30:37 [ReplicaSetMonitorWatcher] starting Wed Feb 22 01:30:37 [Balancer] updated set (shard2) to: shard2/mongodb-shard2a:30021,mongodb-shard2c:30023 Wed Feb 22 01:30:37 [Balancer] updated set (shard2) to: shard2/mongodb-shard2a:30021,mongodb-shard2c:30023,mongodb-shard2b:30022 Wed Feb 22 01:30:37 [Balancer] config servers and shards contacted successfully Wed Feb 22 01:30:37 [Balancer] balancer id: appserver:27017 started at Feb 22 01:30:37 Wed Feb 22 01:30:37 [Balancer] created new distributed lock for balancer on mongodb-config1:27019,mongodb-config2:27019,mongodb-config3:27019 ( lock timeout : 900000, ping interval : 30000, process : 0 ) Wed Feb 22 01:30:37 [Balancer] SyncClusterConnection connecting to [mongodb-config1:27019] Wed Feb 22 01:30:37 [Balancer] SyncClusterConnection connecting to [mongodb-config2:27019] Wed Feb 22 01:30:37 [Balancer] SyncClusterConnection connecting to [mongodb-config3:27019]

SECTION: SCALING OUT ADDING NEW SHARDS

In this example we'll be adding three new servers, these will be combined into a new ReplicaSet and added to the cluster as a new shard. This section shows how the NetApp C-Mode scale-out capabilities are a good match for the MongoDB scale-out sharding features.

Create flexvols cmode81::> volume create -vserver vserver1 -volume mongodb_shard3 -aggregate aggr1 -size 3G -state online -type RW -policy mongodb_shard3 -user root -security-style unix -autosize true -space-guarantee none -unix-permissions 0777 -junction-path /vol/mongodb_shard13

Change snapshot policy

Scheduled snapshots should be disabled, we'll be taking MongoDB consistent snapshots through SnapCreator.

cmode81::> vol modify -volume mongodb_shard3 -snapshot-policy none

Create qtreescmode81::> qtree create -volume mongodb_shard3 -qtree mongodb_shard3a -security-style unix -oplock-mode enable -vserver vserver1

cmode81::> qtree create -volume mongodb_shard3 -qtree mongodb_shard3b -security-style unix -oplock-mode enable -vserver vserver1 cmode81::> qtree create -volume mongodb_shard3 -qtree mongodb_shard3c -security-style unix -oplock-mode enable -vserver vserver1

Define an export-policycmode81::> vserver export-policy create -policyname mongodb_shard3 -vserver vserver1





MONGODB-SHARD3A

Create mongodb user[root@mongodb-shard3a /]# groupadd -g 8000 mongodb [root@mongodb-shard3a /]# useradd -u 8000 -g 8000 -d /data mongodb

Edit /etc/fstab and include the mountpoints for the mongodb datafilesvserver1-nfs:/vol/mongodb_shard3/mongodb_shard3a /data nfs rw,vers=3,rsize=65535,wsize=65535,hard,proto=tcp,timeo=600,nolock,retrans=2,sec=sys,addr=172.16.1.97 0 0

[root@mongodb-shard3a /]# mount /data [root@mongodb-shard3a /]# mkdir -p /data/db/ [root@mongodb-shard3a /]# chmod -R 700 /data [root@mongodb-shard3a /]# chown -R mongodb:mongodb /data

Install mongodb software[root@mongodb-shard3a /]# cd /opt

[root@mongodb-shard3a /]# wget http://fastdl.mongodb.org/linux/mongodb-linux-x86_64-2.0.2.tgz [root@mongodb-shard3a /]# tar xzf mongodb-linux-x86_64-2.0.2.tgz [root@mongodb-shard3a /]# ln -s mongodb-linux-x86_64-2.0.2 mongodb

MONGODB-SHARD3B

Create mongodb user


Edit /etc/fstab and include the mountpoints for the mongodb datafiles vserver1-nfs:/vol/mongodb_shard3/mongodb_shard3b /data nfs rw,vers=3,rsize=65535,wsize=65535,hard,proto=tcp,timeo=600,nolock,retrans=2,sec=sys,addr=172.16.1.98 0 0



[root@mongodb-shard3b /]# cd /opt [root@mongodb-shard3b /]# wget http://fastdl.mongodb.org/linux/mongodb-linux-x86_64-2.0.2.tgz [root@mongodb-shard3b /]# tar xzf mongodb-linux-x86_64-2.0.2.tgz [root@mongodb-shard3b /]# ln -s mongodb-linux-x86_64-2.0.2 mongodb

MONGODB-SHARD3C

Create mongodb user[root@mongodb-shard3c /]# groupadd -g 8000 mongodb [root@mongodb-shard3c /]# useradd -u 8000 -g 8000 -d /data mongodb

Edit /etc/fstab and include the mountpoints for the mongodb datafilesvserver1-nfs:/vol/mongodb_shard3/mongodb_shard3c /data nfs rw,vers=3,rsize=65535,wsize=65535,hard,proto=tcp,timeo=600,nolock,retrans=2,sec=sys,addr=172.16.1.100 0 0


Install mongodb software[root@mongodb-shard3c /]# cd /opt [root@mongodb-shard3c /]# wget http://fastdl.mongodb.org/linux/mongodb-linux-x86_64-2.0.2.tgz [root@mongodb-shard3c /]# tar xzf mongodb-linux-x86_64-2.0.2.tgz [root@mongodb-shard3c /]# ln -s mongodb-linux-x86_64-2.0.2 mongodb

Start replica members

MONGODB-SHARD3A [root@mongodb-shard3a /]# su - mongo [mongodb@mongodb-shard1a /]> /opt/mongodb/bin/mongod --replSet shard3/mongodb-shard3b:30032,mongodb-shard3c:30033 --journal --rest --shardsvr –-port 30031 --fork --logpath /data/db/mongod.log

MONGODB-SHARD3B[root@mongodb-shard3b /]# su - mongo [mongodb@mongodb-shard3b /]> /opt/mongodb/bin/mongod --replSet shard3/mongodb-shard3a:30031,mongodb-shard3c:30033 --journal --rest --shardsvr –-port 30032 --fork --logpath /data/db/mongod.log

MONGODB-SHARD3C[root@mongodb-shard3c /]# su - mongo [mongodb@mongodb-shard3c /]> /opt/mongodb/bin/mongod --replSet shard3/mongodb-shard3a:30031,mongodb-shard3b:30032 --journal --rest --shardsvr –-port 30033 --fork --logpath /data/db/mongod.log

Build replica sets

Connect to application server APPSERVER[mongodb@appserver /]> /opt/mongodb/bin/mongo mongodb-shard3a:30011/admin cfg = { _id : "shard3", members : [ {_id : 0, host : "mongodb-shard3a:30031"}, {_id : 1, host : "mongodb-shard3b:30032"}, {_id : 2, host : "mongodb-shard3c:30033"} ] }

rs.initiate(cfg); exit;

Add shards

Connect to APPSERVER[root@appserver /]> su - mongodb [mongodb@appserver /]> /opt/mongodb/bin/mongo appserver:27017/admin

db.adminCommand( { addShard : "shard3/mongodb-shard3a:30031,mongodb-shard3b:30032,mongodb-shard3c:30033" } )

Verify Shard Information

The shards will re-balance over time as the 'balancer' process works in the background.

mongos> db.printShardingStatus(); --- Sharding Status --- sharding version: { "_id" : 1, "version" : 3 } shards: { "_id" : "shard1", "host" : "shard1/mongodb-shard1c:30013,mongodb-shard1c:30013,mongodb-shard1b:30012,mongodb-shard1a:30011" } { "_id" : "shard2", "host" : "shard2/mongodb-shard2a:30021,mongodb-shard2c:30023,mongodb-shard2b:30022" } { "_id" : "shard3", "host" : "shard3/mongodb-shard3a:30031,mongodb-shard3b:30032,mongodb-shard3c:30033" } databases: { "_id" : "admin", "partitioned" : false, "primary" : "config" } { "_id" : "test", "partitioned" : false, "primary" : "shard1" } { "_id" : "tickets", "partitioned" : true, "primary" : "shard2" } { "_id" : "lottery", "partitioned" : true, "primary" : "shard1" } lottery.tickets chunks: shard1 17 shard2 15 too many chunks to print, use verbose if you want to force print { "_id" : "settings", "partitioned" : false, "primary" : "shard2" } { "_id" : "appserver", "partitioned" : false, "primary" : "shard2" }

mongos>

SECTION: RUNNING HADOOP MAP/REDUCE ON DEMAND

The section ”database cloning for dev/test environments” earlier in this document shows how NetApp flexclones can be used to quickly build a dev/test environment or to assist in partial database restores.

The same approach used in those examples can be used to build an hadoop environment on demand. A group of physical or virtualised servers could be re-purposed temporarly to run a map/reduce job and discarded as soon the batch job is complete.

In the example that follows, a set of servers was allocated for running a cloned MongoDB landscape and another group of servers allocated for running the Hadoop Map/Reduce tasks.

MongoDB provides a hadoop plugin which allows for Hadoop to collect map/reduce data directly from a MongoDB collection and insert the map/reduce job results into another collection of the same/different MongoDB database.

By using the same source of data as input for Hadoop Map/Reduce it is simple to leverage flexclones to build a cloned MongoDB instance on the fly, removing the CPU impact of running map/reduce jobs on the MongoDB production landscape by running these jobs against the cloned MongoDB instance.

The map/reduce results can be then stored either on the cloned MongoDB environment or sent directly to the production environment.

The examples provided make use of Cloudera Hadoop 3 Update 4 and the MongoDB hadoop adapter plugin 1.0.1.

Preparation steps

The map/reduce environment, consists of nineservers for the mongodb cloned landscape, and five servers for the hadoop map/reduce jobs.

This is just an example of a possible configuration, another possibility would be for example to reduce the number of mongodb servers or to simply run the mongodb instances on the same servers as the hadoop processes.

Note that the MongoDB Hadoop adapter will not use HDFS for the map/reduce jobs, the hdfs is used solely for temporary data storage required during the stagging process.

The list of hadoop servers (or virtual machines)

• hadoop-namenode (runs namenode)

• hadoop-jobtracker (runs jobtracker)

• hadoop-datanode1,2,3 (runs datanode, tasktracker)

The map/reduce jobs will be submited from the APPSERVER, for this we will have to install Hadoop on this server.

The list of MongoDB servers (or virtual machines)

• mp-mongodb-shard1a

• mp-mongodb-shard1b

• mp-mongodb-shard1c







Create flexvols for the Namenode

Following Hadoop and Cloudera best-practices, the HDFS metadata will be stored on local disk and also on a NFS share.cmode81::> volume create -vserver vserver1 -volume hadoop_namenode -aggregate aggr1 -size 3G -state online -type RW -policy hadoop_namenode -user root -security-style unix -autosize true -space-guarantee none -unix-permissions 0777 -junction-path /vol/hadoop_namenode

Create qtreescmode81::> qtree create -volume hadoop_namenode -qtree dfs_nn -security-style unix -oplock-mode enable -vserver vserver1

Define an export-policycmode81::> vserver export-policy create -policyname hadoop_namenode -vserver vserver1

cmode81::> vserver export-policy rule create -policyname hadoop_namenode -vserver vserver1 -ruleindex 1 -protocol nfs3 -clientmatch hadoop-namenode -rorule any -rwrule any -anon 0 -superuser any

FIX /etc/hosts on all cloned MongoDB servers

As in the dev/test example in this document, the /etc/hosts file needs to be 'fixed' so that the mongodb servers send their queries to the 'correct' cloned mongod instance.172.16.1.108 mp-mongodb-shard1a 172.16.1.109 mp-mongodb-shard1b 172.16.1.110 mp-mongodb-shard1c 172.16.1.111 mp-mongodb-shard2a 172.16.1.112 mp-mongodb-shard2b 172.16.1.113 mp-mongodb-shard2c 172.16.1.114 mp-mongodb-shard3a 172.16.1.115 mp-mongodb-shard3b 172.16.1.116 mp-mongodb-shard3c

Install SnapCreator Agent on all cloned MongoDB servers

The cloned MongoDB instances will be cloned using SnapCreator on all cloned MongoDB servers.

cd /usr/local/scServer3.5.0./snapcreator --profile setup

Copy and extract the software to /usr/local

Base Software EULA v-11-02-11


Setup NetApp Snap Creator Framework 3.5.0 Server (y|n): n

Setup NetApp Snap Creator Framework 3.5.0 Agent (y|n): y

INFO: Updated NetApp Snap Creator Framework 3.5.0 Agent INFO: To start the NetApp Snap Creator Framework 3.5.0 Agent run "/usr/local/scAgent3.5.0/bin/scAgent start" INFO: To stop the NetApp Snap Creator Framework 3.5.0 Agent run "/usr/local/scAgent3.5.0/bin/scAgent stop"

Start the SnapCreator Agent on all cloned MongoDB servers /usr/local/scAgent3.5.0/bin/scAgent start Starting scAgent: Checking Status of scAgent: Running

Configure SC Agent access on all cloned MongoDB servers cat /usr/local/scAgent3.5.0/config/agent.conf ############################################ ### SnapCreator Agent Configuration File ### ############################################ ###################################################################### ### Command to allow or wildcard "*" can be used to allow anything ### ###################################################################### host: scServer@* command: *

Install MongoDB software on all cloned MongoDB serversgroupadd -g 8000 mongodb useradd -u 8000 -g 8000 -d /data mongodb

cd /opt wget http://fastdl.mongodb.org/linux/mongodb-linux-x86_64-2.0.2.tgz tar xzf mongodb-linux-x86_64-2.0.2.tgz ln -s mongodb-linux-x86_64-2.0.2 mongodb

Build SnapCreator Profiles

A limitation in the SnapCreator 3.5 release prevents a SnapCreator config file to execute commands or actions on multiple servers as each SnapCreator config file is limited to one agent.

In order to build the cloned mongod databases and a flexcloned copy of its original production flexvols, it will be necessary to create one config file for each server.

• mongodb/mp_mongodb_shard1a.conf • mongodb/mp_mongodb_shard1b.conf • mongodb/mp_mongodb_shard1c.conf • mongodb/mp_mongodb_shard2a.conf • mongodb/mp_mongodb_shard2b.conf • mongodb/mp_mongodb_shard2c.conf • mongodb/mp_mongodb_shard3a.conf • mongodb/mp_mongodb_shard3b.conf • mongodb/mp_mongodb_shard3c.conf

LOGIN TO appserver

Copy SnapCreator default.conf config to mongodb/mp_mongodb_shard1a.conf and edit the following sections

#################################################### Snap Creator Framework Configuration File ####################################################

########################### Required Options ###########################CONFIG_TYPE=STANDARDSNAME=MONGODBSNAP_TIMESTAMP_ONLY=NVOLUMES=vserver1-mgmt:mongodb_shard1NTAP_SNAPSHOT_RETENTIONS=daily:2NTAP_USERS=vserver1-mgmt:snapcreator/netapp123NTAP_PWD_PROTECTION=NTRANSPORT=HTTPPORT=80LOG_NUM=30LOG_TRACE_ENABLE=YSC_TMP_DIR=


########################

### Snapshot Options ###########################NTAP_SNAPSHOT_RETENTION_AGE=SNAPDRIVE=NSNAPDRIVE_DISCOVERY=NNTAP_SNAPSHOT_DISABLE=YNTAP_SNAPSHOT_NODELETE=YNTAP_CONSISTENCY_GROUP_SNAPSHOT=NNTAP_CONSISTENCY_GROUP_TIMEOUT=mediumNTAP_CONSISTENCY_GROUP_WAFL_SYNC=NNTAP_SNAPSHOT_DELETE_BY_AGE_ONLY=NNTAP_SNAPSHOT_DEPENDENCY_IGNORE=NNTAP_SNAPSHOT_RESTORE_AUTO_DETECT=N

########################## Cloning Options ##########################NTAP_VOL_CLONE_RESERVE=noneNTAP_LUN_CLONE_RESERVATION=falseNTAP_CLONE_IGROUP_MAP=NTAP_CLONE_FOR_BACKUP=YNTAP_CLONE_SECONDARY=NNTAP_CLONE_SECONDARY_VOLUMES=NTAP_NUM_VOL_CLONES=1NTAP_NFS_EXPORT_HOST=mp-mongodb-shard1aNTAP_NFS_EXPORT_ACCESS=read-writeNTAP_NFS_EXPORT_PERSISTENT=false

####################### APP Commands #######################ARCHIVE_CMD01=UMOUNT_CMD01=AGENT:killall -u mongodb ; exit 0UMOUNT_CMD02=AGENT:umount /data ; exit 0

MOUNT_CMD01=AGENT:mount -t nfs -o rw,vers=3,rsize=32768,wsize=32768,hard,proto=tcp,timeo=600,retrans=2,sec=sys,addr=172.16.1.104 vserver1-nfs:/cl_mp_mongodb_shard1a_mongodb_shard1_%SNAP_TIME/mongodb_shard1a /data

MOUNT_CMD10=AGENT:chown -R mongodb:mongodb /data/* ; exit 0MOUNT_CMD11=AGENT:rm -f /data/db/mongod.lockMOUNT_CMD12=AGENT:su - mongodb -c '/opt/mongodb/bin/mongod --replSet shard1/mongodb-shard1b:30012,mongodb-shard1c:30013 --journal --rest --shardsvr --port 30011 --dbpath /data/db --fork --logpath /data/db/mongod.log'

MOUNT_CMD21=AGENT:su - mongodb -c '/opt/mongodb/bin/mongod --configsvr --port 4011 --dbpath /data/configdb/ --fork --logpath /data/configdb/mongod.log'

MOUNT_CMD25=AGENT:su - mongodb -c 'sleep 60; /opt/mongodb/bin/mongos --configdb mp-mongodb-shard1a:4011,mp-mongodb-shard2a:4021,mp-mongodb-shard2c:4023 --fork --logpath /data/db/mongos.log'

############################## Event Configuration ##############################NTAP_ASUP_ERROR_ENABLE=NSENDTRAP=SUCCESS_TRAP=SUCCESS_MSG=INFO: Snap Creator finished successfully ( Action: %ACTION )

###################################### Client/Server Configuration ######################################SC_AGENT=mp-mongodb-shard1a:9090SC_AGENT_TIMEOUT=120SC_AGENT_UNQUIESCE_TIMEOUT=SC_CLONE_TARGET=mp-mongodb-shard1a:9090SC_AGENT_WATCHDOG_ENABLE=YSC_AGENT_LOG_ENABLE=Y

Copy SnapCreator default.conf config to mongodb/mp_mongodb_shard1b.conf and edit the following sections

#################################################

### Snap Creator Framework Configuration File ####################################################



########################### Snapshot Options ###########################NTAP_SNAPSHOT_RETENTION_AGE=SNAPDRIVE=NSNAPDRIVE_DISCOVERY=NNTAP_SNAPSHOT_DISABLE=YNTAP_SNAPSHOT_NODELETE=YNTAP_CONSISTENCY_GROUP_SNAPSHOT=NNTAP_CONSISTENCY_GROUP_TIMEOUT=mediumNTAP_CONSISTENCY_GROUP_WAFL_SYNC=NNTAP_SNAPSHOT_DELETE_BY_AGE_ONLY=NNTAP_SNAPSHOT_DEPENDENCY_IGNORE=NNTAP_SNAPSHOT_RESTORE_AUTO_DETECT=N

########################## Cloning Options ##########################NTAP_VOL_CLONE_RESERVE=noneNTAP_LUN_CLONE_RESERVATION=falseNTAP_CLONE_IGROUP_MAP=NTAP_CLONE_FOR_BACKUP=YNTAP_CLONE_SECONDARY=NNTAP_CLONE_SECONDARY_VOLUMES=NTAP_NUM_VOL_CLONES=1NTAP_NFS_EXPORT_HOST=mp-mongodb-shard1bNTAP_NFS_EXPORT_ACCESS=read-writeNTAP_NFS_EXPORT_PERSISTENT=false


MOUNT_CMD02=AGENT:mount -t nfs -o rw,vers=3,rsize=32768,wsize=32768,hard,proto=tcp,timeo=600,retrans=2,sec=sys,addr=172.16.1.104 vserver1-nfs:/cl_mp_mongodb_shard1b_mongodb_shard1_%SNAP_TIME/mongodb_shard1b /data

MOUNT_CMD10=AGENT:chown -R mongodb:mongodb /data/* ; exit 0MOUNT_CMD11=AGENT:rm -f /data/db/mongod.lock

MOUNT_CMD13=AGENT:su - mongodb -c '/opt/mongodb/bin/mongod --replSet shard1/mongodb-shard1a:30011,mongodb-shard1c:30013 --journal --rest --shardsvr --port 30012 --dbpath /data/db --fork --logpath /data/db/mongod.log'



###################################### Client/Server Configuration ######################################SC_AGENT=mp-mongodb-shard1b:9090SC_AGENT_TIMEOUT=120SC_AGENT_UNQUIESCE_TIMEOUT=SC_CLONE_TARGET=mp-mongodb-shard1b:9090SC_AGENT_WATCHDOG_ENABLE=YSC_AGENT_LOG_ENABLE=Y

Copy SnapCreator default.conf config to mongodb/mp_mongodb_shard1c.conf and edit the following sections


################################ Snap Creator Settings ################################




########################## Cloning Options ##########################NTAP_VOL_CLONE_RESERVE=noneNTAP_LUN_CLONE_RESERVATION=falseNTAP_CLONE_IGROUP_MAP=NTAP_CLONE_FOR_BACKUP=Y

NTAP_CLONE_SECONDARY=NNTAP_CLONE_SECONDARY_VOLUMES=NTAP_NUM_VOL_CLONES=1NTAP_NFS_EXPORT_HOST=mp-mongodb-shard1cNTAP_NFS_EXPORT_ACCESS=read-writeNTAP_NFS_EXPORT_PERSISTENT=false


MOUNT_CMD03=AGENT:mount -t nfs -o rw,vers=3,rsize=32768,wsize=32768,hard,proto=tcp,timeo=600,retrans=2,sec=sys,addr=172.16.1.104 vserver1-nfs:/cl_mp_mongodb_shard1c_mongodb_shard1_%SNAP_TIME/mongodb_shard1c /data


MOUNT_CMD14=AGENT:su - mongodb -c '/opt/mongodb/bin/mongod --replSet shard1/mongodb-shard1a:30011,mongodb-shard1b:30012 --journal --rest --shardsvr --port 30013 --dbpath /data/db --fork --logpath /data/db/mongod.log'



###################################### Client/Server Configuration ############################################SC_AGENT=mp-mongodb-shard1c:9090SC_AGENT_TIMEOUT=120SC_AGENT_UNQUIESCE_TIMEOUT=SC_CLONE_TARGET=mp-mongodb-shard1c:9090SC_AGENT_WATCHDOG_ENABLE=YSC_AGENT_LOG_ENABLE=Y










MOUNT_CMD15=AGENT:su - mongodb -c '/opt/mongodb/bin/mongod --replSet shard2/mongodb-shard2b:30022,mongodb-shard2c:30023 --journal --rest --shardsvr --port 30021 --dbpath /data/db --fork --logpath /data/db/mongod.log'

MOUNT_CMD22=AGENT:su - mongodb -c '/opt/mongodb/bin/mongod --configsvr --port 4021 --dbpath /data/configdb/ --fork --logpath /data/configdb/mongod.log'MOUNT_CMD25=AGENT:su - mongodb -c 'sleep 60; /opt/mongodb/bin/mongos --configdb mp-mongodb-shard1a:4011,mp-mongodb-shard2a:4021,mp-mongodb-shard2c:4023 --fork --logpath /data/db/mongos.log'


###################################### Client/Server Configuration ############################################SC_AGENT=mp-mongodb-shard2a:9090SC_AGENT_TIMEOUT=120SC_AGENT_UNQUIESCE_TIMEOUT=SC_CLONE_TARGET=mp-mongodb-shard2a:9090SC_AGENT_WATCHDOG_ENABLE=YSC_AGENT_LOG_ENABLE=Y

Copy SnapCreator default.conf config to mongodb/mp_mongodb_shard2b.conf and edit the following sections










MOUNT_CMD25=AGENT:su - mongodb -c 'sleep 60; /opt/mongodb/bin/mongos --configdb mp-mongodb-

shard1a:4011,mp-mongodb-shard2a:4021,mp-mongodb-shard2c:4023 --fork --logpath /data/db/mongos.log'


###################################### Client/Server Configuration ############################################SC_AGENT=mp-mongodb-shard2b:9090SC_AGENT_TIMEOUT=120SC_AGENT_UNQUIESCE_TIMEOUT=SC_CLONE_TARGET=mp-mongodb-shard2b:9090SC_AGENT_WATCHDOG_ENABLE=YSC_AGENT_LOG_ENABLE=Y






########################## Cloning Options ##########################NTAP_VOL_CLONE_RESERVE=noneNTAP_LUN_CLONE_RESERVATION=falseNTAP_CLONE_IGROUP_MAP=NTAP_CLONE_FOR_BACKUP=YNTAP_CLONE_SECONDARY=NNTAP_CLONE_SECONDARY_VOLUMES=NTAP_NUM_VOL_CLONES=1

NTAP_NFS_EXPORT_HOST=mp-mongodb-shard2cNTAP_NFS_EXPORT_ACCESS=read-writeNTAP_NFS_EXPORT_PERSISTENT=false





MOUNT_CMD23=AGENT:su - mongodb -c '/opt/mongodb/bin/mongod --configsvr --port 4023 --dbpath /data/configdb/ --fork --logpath /data/configdb/mongod.log'



###################################### Client/Server Configuration ##################################################################################################################################################SC_AGENT=mp-mongodb-shard2c:9090SC_AGENT_TIMEOUT=120SC_AGENT_UNQUIESCE_TIMEOUT=SC_CLONE_TARGET=mp-mongodb-shard2c:9090SC_AGENT_WATCHDOG_ENABLE=YSC_AGENT_LOG_ENABLE=Y











MOUNT_CMD18=AGENT:su - mongodb -c '/opt/mongodb/bin/mongod --replSet shard3/mongodb-shard3b:30032,mongodb-shard3c:30033 --journal --rest --shardsvr --port 30031 --dbpath /data/db --fork --logpath /data/db/mongod.log'



###################################### Client/Server Configuration ############################################SC_AGENT=mp-mongodb-shard3a:9090SC_AGENT_TIMEOUT=120SC_AGENT_UNQUIESCE_TIMEOUT=SC_CLONE_TARGET=mp-mongodb-shard3a:9090SC_AGENT_WATCHDOG_ENABLE=YSC_AGENT_LOG_ENABLE=Y

Copy SnapCreator default.conf config to mongodb/mp_mongodb_shard3b.conf and edit the following sections #################################################### Snap Creator Framework Configuration File ####################################################












###################################### Client/Server Configuration ############################################SC_AGENT=mp-mongodb-shard3b:9090SC_AGENT_TIMEOUT=120SC_AGENT_UNQUIESCE_TIMEOUT=SC_CLONE_TARGET=mp-mongodb-shard3b:9090SC_AGENT_WATCHDOG_ENABLE=YSC_AGENT_LOG_ENABLE=Y






########################## Cloning Options ###

#######################NTAP_VOL_CLONE_RESERVE=noneNTAP_LUN_CLONE_RESERVATION=falseNTAP_CLONE_IGROUP_MAP=NTAP_CLONE_FOR_BACKUP=YNTAP_CLONE_SECONDARY=NNTAP_CLONE_SECONDARY_VOLUMES=NTAP_NUM_VOL_CLONES=1NTAP_NFS_EXPORT_HOST=mp-mongodb-shard3cNTAP_NFS_EXPORT_ACCESS=read-writeNTAP_NFS_EXPORT_PERSISTENT=false







###################################### Client/Server Configuration ############################################SC_AGENT=mp-mongodb-shard3c:9090SC_AGENT_TIMEOUT=120SC_AGENT_UNQUIESCE_TIMEOUT=SC_CLONE_TARGET=mp-mongodb-shard3c:9090SC_AGENT_WATCHDOG_ENABLE=YSC_AGENT_LOG_ENABLE=Y

Build SnapCreator Clone From Snapshot /usr/local/scServer3.5.0/snapcreator --profile mongodb --config mp_mongodb_shard1a --action clone_vol --snap_name MONGODB-daily_recent --policy daily --clone_name mongodb_mapreduce --verbose

/usr/local/scServer3.5.0/snapcreator --profile mongodb --config mp_mongodb_shard1b --action clone_vol --snap_name MONGODB-daily_recent --policy daily --clone_name mongodb_mapreduce --verbose

/usr/local/scServer3.5.0/snapcreator --profile mongodb --config mp_mongodb_shard1c --action clone_vol --snap_name MONGODB-daily_recent --policy daily --clone_name mongodb_mapreduce --verbose

/usr/local/scServer3.5.0/snapcreator --profile mongodb --config mp_mongodb_shard2a --action clone_vol --snap_name MONGODB-daily_recent --policy daily --clone_name mongodb_mapreduce --verbose



/usr/local/scServer3.5.0/snapcreator --profile mongodb --config mp_mongodb_shard3a --action clone_vol --snap_name MONGODB-daily_recent --policy daily --clone_name mongodb_mapreduce --verbose



Install Haddop software on hadoop nodes and APPSERVER

As the APPSERVER will be used to submit jobs to the hadoop cluster, it is required to install some hadoop packages on this server.

Cloudera has some recommendations regarding Java versions to use with Cloudera Hadoop, as per Cloudera best-practices we will be using JDK 1.6.0.26.

Download and install this java version across all hadoop nodes and the APPSERVER.

Download JDK 1.6.0.26 from: http://www.oracle.com/technetwork/java/javasebusiness/downloads/java-archive-downloads-javase6-419409.html

and install it under /opt/

Set JAVA_HOME and PATH on /etc/profile export JAVA_HOME=/opt/jdk1.6.0_26/ export PATH=$JAVA_HOME/bin:$PATH

Install hadoop packages as per cloudera instructions: https://ccp.cloudera.com/display/CDHDOC/CDH3+Installation#CDH3Installation-InstallingCDH3OnRedHatcompatiblesystems

wget http://archive.cloudera.com/redhat/6/x86_64/cdh/cdh3-repository-1.0-1.noarch.rpm

yum --nogpgcheck localinstall cdh3-repository-1.0-1.noarch.rpm rpm --import http://archive.cloudera.com/redhat/6/x86_64/cdh/RPM-GPG-KEY-cloudera yum search hadoop yum install hadoop-0.20 hadoop-0.20-native

Install hadoop specific packages

on hadoop-namenode: yum install hadoop-0.20-namenode

on hadoop-jobtracker: yum install hadoop-0.20-jobtracker

on hadoop-datanode: yum install hadoop-0.20-tasktracker yum install hadoop-0.20-datanode

Configure hadoop

on the namenode:

edit /etc/conf/hadoop/hadoop-env.sh

set: export JAVA_HOME=/opt/jdk1.6.0_26

Edit /etc/fstab and include the mountpoints for the NFS volume

https://ccp.cloudera.com/display/CDHDOC/CDH3+Installation#CDH3Installation-InstallingCDH3OnRedHatcompatiblesystems


http://www.oracle.com/technetwork/java/javasebusiness/downloads/java-archive-downloads-javase6-419409.html

http://www.oracle.com/technetwork/java/javasebusiness/downloads/java-archive-downloads-javase6-419409.html

vserver1-nfs:/vol/hadoop-namenode/dfs_nn /nfsmount nfs rw,vers=3,rsize=65535,wsize=65535,hard,proto=tcp,timeo=600,nolock,retrans=2,sec=sys,addr=172.16.1.103 0 0

mount /nfsmountmkdir -p /nfsmount/dfs/nnmkdir -p /data/1/dfs/nnchown -R hdfs:hdfs /nfsmount /data

edit core-site.xml

<configuration> <property> <name>fs.default.name</name> <value>hdfs://hadoop-namenode:9000</value> </property> </configuration>

edit hdfs-site.xml

<property> <name>dfs.name.dir</name> <value>/data/1/dfs/nn,/nfsmount/dfs/nn</value> </property> <property>

edit mapred-site.xml

<configuration> <property> <name>mapred.job.tracker</name> <value>hadoop-jobtracker:9001</value> </property> </configuration>

on the jobtracker:

edit /etc/conf/hadoop/hadoop-env.sh set:

export JAVA_HOME=/opt/jdk1.6.0_26

edit core-site.xml




on datanodes1,2,3

edit /etc/conf/hadoop/hadoop-env.sh

set:

export JAVA_HOME=/opt/jdk1.6.0_26

mkdir -p /data/1/dfs/dn /data/2/dfs/dn /data/3/dfs/dn chown -R hdfs:hdfs /data

edit core-site.xml


edit hdfs-site.xml <property> <name>dfs.data.dir</name> <value>/data/1/dfs/dn,/data/2/dfs/dn,/data/3/dfs/dn</value> </property>



on the appserver:

edit /etc/conf/hadoop/hadoop-env.sh set: export JAVA_HOME=/opt/jdk1.6.0_26

edit core-site.xml




Format HDFS sudo -u hdfs hadoop namenode -format sudo -u hdfs hadoop chmod 777 / sudo -u hdfs hadoop fs -mkdir /tmp sudo -u hdfs hadoop fs -chmod -R 1777 /tmp sudo -u hdfs hadoop fs -mkdir /mapred/system sudo -u hdfs hadoop fs -chown mapred:hadoop /mapred/system

Start the HADOOP processes

On the hadoop-namenode server, start the namenode processes /etc/rc.d/init.d/hadoop-0.20-namenode start

On the hadoop-datanodes servers, start the datanode and tasktracker processes /etc/rc.d/init.d/hadoop-0.20-datanodes start /etc/rc.d/init.d/hadoop-0.20-tasktracker start

On the hadoop-jobtracker server, start the jobtracker processes /etc/rc.d/init.d/hadoop-0.20-jobtracker start

Install mongodb plugin for hadoop

login to the APPSERVER

Download and install the MongoDB JAVA Driverwget --no-check-certificate https://github.com/downloads/mongodb/mongo-java-driver/mongo-2.7.3.jar $ cp mongo-2.7.3.jar /usr/lib/hadoop/lib/

download the MongoDB-Hadoop Adapter source yum install git git clone https://github.com/mongodb/mongo-hadoop.git

git checkout release-1.0

edit the sources

[root@hadoop-namenode hadoop-plugin]# cat mongo-hadoop/build.sbt name := "mongo-hadoop" organization := "org.mongodb" hadoopRelease in ThisBuild := "cdh3"

Build the adapter ./sbt package cd streaming ./build.sh $ cp core/target/mongo-hadoop-core_cdh3u3-1.0.0.jar /usr/lib/hadoop/lib/

$cp target/mongo-hadoop-streaming-assembly-1.0.1.jar /usr/lib/hadoop/lib/

Install the .jar files across all hadoop servers

Copy the generated files to /usr/lib/hadoop/lib on all haddop servers

mongo-2.7.3.jarcore/target/mongo-hadoop-core_cdh3u3-1.0.0.jartarget/mongo-hadoop-streaming-assembly-1.0.1.jar

install python pymongo_hadoop on all your hadoop nodes and the APPSERVER

sudo yum install gcc python-devel sudo yum install python-setuptools easy_install pymongo_hadoop

https://github.com/downloads/mongodb/mongo-java-driver/mongo-2.7.3.jar

Running a MAP/REDUCE job

We now have all the software in place to submit a hadoop map/reduce job collecting data from our cloned MongoDB and reducing the results back into the same database.

The cloned MongoDB database contains the lottery.tickets collection, with the following objects: mongos> db.tickets.findOne() { "_id" : ObjectId("4f3af764466d6f1a1c000000"), "lnhash" : NumberLong("3667378176"), "week" : 1, "agency" : 927, "luckynumbers" : [ 33, 24, 4, 38, 32, 4, 14, 17 ], "ticketnumber" : 836626 } mongos>

Using the simulated lottery data from the examples in this book, we will build a map/reduce job that computes the most chosen lucky numbers across the lottery tickets.

Our map function will map the members of 'luckynumbers array' and the reduce will sum all occorrences of those numbers allowing us to find out which are the most used 'luckynumbers'.

Copy the Mapper function script to mapper.py on the APPSERVER$ cat mapper.py #!/usr/bin/env python2 import sys sys.path.append(".") from pymongo_hadoop import BSONMapper def mapper(documents): for doc in documents: for number in doc['luckynumbers']: yield{'_id':number,'count':1}

BSONMapper(mapper) print >>sys.stderr, "Done Mapping."

Copy the Reducer function script to reducer.py on the APPSERVER$ cat reducer.py #!/usr/bin/env python2 import sys sys.path.append(".") from pymongo_hadoop import BSONReducer def reducer(key,values):

print >> sys.stderr,"number %s" % key _count = 0 for v in values: _count += v['count'] return {'_id': key, 'count': _count}

BSONReducer(reducer)

RUN MAP/REDUCE JOB

To run the hadoop map/reduce job, login to the APP server and execute the following command line.

Haddop will invoke the mongo-hadoop adapter, the mongo-haddop-streaming-assembly.jar file provides streaming support for reading and writing directly to mongoDB and the hability to use python (pymongo) map/reduce scripts.

[root@appserver scServer3.5.0]# sudo -u hdfs hadoop jar /usr/lib/hadoop-0.20/lib/mongo-hadoop-streaming-assembly-1.0.1.jar -files mapper.py,reducer.py -mapper mapper.py -reducer reducer.py -inputURI mongodb://mp-mongodb-shard1a:27017/lottery.tickets -outputURI mongodb://mp-mongodb-shard1a:27017/lottery.bestnumbers -verbose

12/06/30 10:22:51 INFO streaming.MongoStreamJob: Running 12/06/30 10:22:51 INFO streaming.MongoStreamJob: Init 12/06/30 10:22:51 INFO streaming.MongoStreamJob: Process Args 12/06/30 10:22:51 INFO streaming.StreamJobPatch: Setup Options' 12/06/30 10:22:51 INFO streaming.StreamJobPatch: PreProcess Args 12/06/30 10:22:51 INFO streaming.StreamJobPatch: Parse Options 12/06/30 10:22:51 INFO streaming.StreamJobPatch: Arg: '-mapper' 12/06/30 10:22:51 INFO streaming.StreamJobPatch: Arg: 'mapper.py' 12/06/30 10:22:51 INFO streaming.StreamJobPatch: Arg: '-reducer' 12/06/30 10:22:51 INFO streaming.StreamJobPatch: Arg: 'reducer.py' 12/06/30 10:22:51 INFO streaming.StreamJobPatch: Arg: '-inputURI' 12/06/30 10:22:51 INFO streaming.StreamJobPatch: Arg: 'mongodb://mp-mongodb-shard1a:27017/lottery.tickets' 12/06/30 10:22:51 INFO streaming.StreamJobPatch: Arg: '-outputURI' 12/06/30 10:22:51 INFO streaming.StreamJobPatch: Arg: 'mongodb://mp-mongodb-shard1a:27017/lottery.bestnumbers' 12/06/30 10:22:51 INFO streaming.StreamJobPatch: Arg: '-numReduceTasks' 12/06/30 10:22:51 INFO streaming.StreamJobPatch: Arg: '3' 12/06/30 10:22:51 INFO streaming.StreamJobPatch: Arg: '-verbose' 12/06/30 10:22:51 INFO streaming.StreamJobPatch: Arg: '-debug' 12/06/30 10:22:51 INFO streaming.StreamJobPatch: Add InputSpecs 12/06/30 10:22:51 INFO streaming.StreamJobPatch: Setup output_ 12/06/30 10:22:51 INFO streaming.StreamJobPatch: Post Process Args STREAM: addTaskEnvironment= ...12/06/30 14:51:39 INFO streaming.StreamJob: map 100% reduce 69% 12/06/30 14:51:53 INFO streaming.StreamJob: map 100% reduce 70% 12/06/30 14:52:25 INFO streaming.StreamJob: map 100% reduce 71% 12/06/30 14:52:46 INFO streaming.StreamJob: map 100% reduce 72% 12/06/30 14:53:09 INFO streaming.StreamJob: map 100% reduce 73% 12/06/30 14:53:26 INFO streaming.StreamJob: map 100% reduce 74% 12/06/30 14:53:52 INFO streaming.StreamJob: map 100% reduce 75% 12/06/30 14:54:09 INFO streaming.StreamJob: map 100% reduce 76% 12/06/30 14:54:19 INFO streaming.StreamJob: map 100% reduce 77% 12/06/30 14:54:49 INFO streaming.StreamJob: map 100% reduce 78% 12/06/30 14:55:02 INFO streaming.StreamJob: map 100% reduce 79% 12/06/30 14:55:36 INFO streaming.StreamJob: map 100% reduce 80% 12/06/30 14:55:45 INFO streaming.StreamJob: map 100% reduce 81% 12/06/30 14:56:16 INFO streaming.StreamJob: map 100% reduce 82% 12/06/30 14:56:30 INFO streaming.StreamJob: map 100% reduce 83% 12/06/30 14:56:57 INFO streaming.StreamJob: map 100% reduce 84% 12/06/30 14:57:13 INFO streaming.StreamJob: map 100% reduce 85% 12/06/30 14:57:20 INFO streaming.StreamJob: map 100% reduce 86% 12/06/30 14:57:56 INFO streaming.StreamJob: map 100% reduce 87% 12/06/30 14:58:03 INFO streaming.StreamJob: map 100% reduce 88% 12/06/30 14:58:42 INFO streaming.StreamJob: map 100% reduce 89% 12/06/30 14:58:44 INFO streaming.StreamJob: map 100% reduce 90% 12/06/30 14:59:23 INFO streaming.StreamJob: map 100% reduce 91% 12/06/30 14:59:26 INFO streaming.StreamJob: map 100% reduce 92% 12/06/30 15:00:07 INFO streaming.StreamJob: map 100% reduce 93% 12/06/30 15:00:12 INFO streaming.StreamJob: map 100% reduce 94% 12/06/30 15:00:50 INFO streaming.StreamJob: map 100% reduce 95% 12/06/30 15:00:56 INFO streaming.StreamJob: map 100% reduce 96% 12/06/30 15:01:33 INFO streaming.StreamJob: map 100% reduce 97% 12/06/30 15:01:39 INFO streaming.StreamJob: map 100% reduce 98%

12/06/30 15:01:41 INFO streaming.StreamJob: map 100% reduce 99% 12/06/30 15:02:04 INFO streaming.StreamJob: map 100% reduce 100% 12/06/30 15:02:13 INFO streaming.StreamJob: Job complete: job_201206300442_0002 12/06/30 15:02:13 INFO streaming.StreamJob: Output: file:///tmp

Verify the results, there should be a new collection in the lottery database.[mongodb@centos ~]$ /opt/mongodb/bin/mongo centos:27017/admin MongoDB shell version: 2.0.2 connecting to: centos:27017/admin

mongos> show dbs admin 0.203125GB config 0.1875GB lottery 9.15625GB test 0.203125GB

mongos> use lottery switched to db lottery

mongos> show collections bestnumbers system.indexes tickets

mongos> db.bestnumbers.find({}).sort({count:-1}).limit(6) { "_id" : 2, "count" : 1887865 } { "_id" : 11, "count" : 1886713 } { "_id" : 14, "count" : 1886464 } { "_id" : 6, "count" : 1886379 } { "_id" : 12, "count" : 1886276 } { "_id" : 15, "count" : 1886270 }

mongos>

CONCLUSION

The content in this document should provide a reader with enough information to set up a MongoDB distributed and highly-available database on NetApp Cluster-Mode.

The 10Gen MongoDB Sharding and ReplicaSet features are a perfect match for Ontap 8 Scale-Out Cluster-Mode, enabling fast 'schema' changes in a DEV/TEST environment with a minum amount of hardware resources. Ontap features such as vol autosize, storage effiency simplify the management of disk allocation to the MongoDB environment.

By leveraging snapshots for backup and restore, it is possible to quickly restore a multi-node MongoDB database in a matter of minutes, or to provide a full cloned landscape where Hadoop Map-Reduce jobs can be run without impacting the production environment.

REFERENCES

http://www.mongodb.org/display/DOCS/Home http://www.mongodb.org/pages/viewpage.action?pageId=21268863&navigatingVersions=true http://www.slideshare.net/spf13/mongodb-hadoop-and-humongous-data http://www.mongodb.org/display/DOCS/Hadoop+Quick%20Start https://ccp.cloudera.com/display/CDHDOC/CDH3+Installation#CDH3Installation-InstallingCDH3OnRedHatcompatiblesystems



http://www.mongodb.org/display/DOCS/Hadoop+Quick%20Start

http://www.slideshare.net/spf13/mongodb-hadoop-and-humongous-data


http://www.mongodb.org/display/DOCS/Home

Documents

BIGDATA - NetAppcommunity.netapp.com/fukiw75442/attachments/fukiw75442/... · 2014-09-24 · BIGDATA SCALING OUT MONGODB and Hadoop Map/Reduce (on demand) ON NETAPP CLUSTER-MODE 8.1