Best Practices for Upgrading to MongoDB 3.2

Richard Kreuter MongoDB

• Upgrading the database itself is pretty easy.• Upgrading all the bits around the database is…

less easy.• The effort is worth it.

• We can help you.

What’s cool in 3.2

More use cases. Pluggable storage engines enables you to use MongoDB in more

projects with the same core database.

Mission-critical apps. MongoDB delivers major advances in the critical areas of

governance, high availability, and disaster recovery.

New tools for new users. Now MongoDB is an integral part of the tooling and

workflows of Data Analysts, DBAs, and Operations teams.

3.2 Overview

Document Validation

What you get • Implement data governance • Enforce data quality• Use familiar MongoDB queries to specify validation rules

More power to the DBAs • The DBA can specify which documents in a collection should be validated

– Pre-existing documents can be left unvalidated, optionally.• Validation failure consequences are tunable

• Hard error• Just a warning

Document Validation Example

The example on the left adds a rule to the

contacts collection that validates:

• The year of birth is no later than 1994

• The document contains a phone number and / or

an email address

• When present, the phone number and email

addresses are strings

Partial Indexes

{"_id" : 1234,"archived" : "true"... }

A partial index canignore all documentswhere ignoredis true.

$lookup Join

What you get • Equivalent to left outer join• Built into aggregation framework

$lookup: { localField : "stock", foreignField : "stockQuote", from : "other_collection", as : "joined_stock" }

Distributed Database Enhancements

Faster Failover for Replica Sets

• Enhanced algorithm detects failure and isolation of primary.• Clusters more resilient to overloaded or unreliable networks

What you get • <2 seconds failover (tunable with the electionTimeoutMillis)

How to upgrade:• Conceptually:

– cfg=rs.conf()– cfg.protocolVersion=1– rs.reconfig(cfg)

Config Servers as Replica Sets

Overview• Config server replica sets can span more than 3 data centers with up to 50 replica set

members supported

What you get • Simplified sharded deployments

– Config servers are deployed as replica sets• Improved metadata consistency • Easily scale to many data centers How to upgrade• New clusters only, or a cluster reboot (for now)

Storage Engines

Storage Engine Architecture in 3.2

Content Repo

IoT Sensor Backend Ad Service Customer

Analytics Archive

MongoDB Query Language (MQL) + Native Drivers

MongoDB Document Data Model

WT MMAP

Available for MongoDB 3.2

Man

agem

ent

Sec

urity

In-memory (beta) Encrypted 3rd party

WiredTiger is the Default

What you get • Best general-purpose storage engine• 7-10x better throughput• Up to 80% compression

“Out of the box” • No special configuration needed

7x-10x Performance, 50%-80% Less Storage

How: WiredTiger Storage Engine• Same data model, same query

language, same ops• Write performance gains driven

by document-level concurrency control

• Storage savings driven by native compression

• 100% backwards compatible• Non-disruptive upgrade

MongoDB 3.0MongoDB 2.6

Performance

50%-80% Less Storage via Compression

• Better storage utilization• Higher I/O scalability• Multiple compression options

– Snappy– zlib– None

• Data and journal compressed on disk• Indexes compressed on disk and in memory

Compression in action

Encrypted Storage

What you get • Encryption of sensitive data for regulated industries• ~ 15% overhead

• Better than many 3rd party encryption tools

Note• Based on the WiredTiger Storage Engine • Available in MongoDB Enterprise Advanced

How to upgrade• Rolling upgrade

In-Memory Storage

• Predictable throughput and latency• In-memory computing without trading away guarantees of disk-based

databases like – Rich query flexibility – Real-time analytics– Scalable capacity – Durability

• Durability via replica set secondaries (using mmap/wt)

One Deployment Powering Multiple Apps

New Tools for New Users

Data Analysts• BI Connector• $lookup

• Aggregation Enhancements: Faster & simpler real-time analytics

DBAsMongoDB Compass

• Visualize schema• View Documents• Graphically build queries• Authenticated access

OperationsIntegration with standard operational workflow reduces overhead• Start from global view of infrastructure: APM tools

integration, e.g. New Relic, AppDynamics.• Then, drill down: Profiler visualization in Ops Manager • Then, deploy: Automated index builds• Next, refine: Partial Indexes improve resource utilization

Ops Manager & Cloud Manager

Single-click provisioning, scaling & upgrades, admin tasks

Monitoring, with charts, dashboards and alerts on 100+ metrics

Backup and restore, with point-in-time recovery, support for sharded clusters

MongoDB Ops Manager / Cloud Manager

The Best Way to Manage MongoDB In Your Data CenterUp to 95% Reduction in Operational Overhead

How Ops Manager / Cloud Manager Helps You

Scale EasilyMeet SLAs

Best Practices, Automated

Cut Management Overhead

Query Perf. Visualizations & Optimization

Fast and simple query optimization with the

new Visual Query Profiler

• Query and write latency are consolidated and

displayed visually; your ops teams can easily

identify slower queries and latency spikes

• Visual query profiler analyzes the data it

displays and provides recommendations for

new indexes that can be created to improve

query performance

• Ops Manager and Cloud Manager can automate

the rollout of new indexes, reducing risk and

your team’s operational overhead

Ops Manager Backup Enhancements3.2 includes Ops Manager enhancements to improve

the productivity of your ops teams and further

simplify installation and management

• MongoDB backup on standard network-mountable filesystems;

integrates with your existing storage infrastructure

• Automated database restores; Build clusters from backup in a few

clicks

• Faster time to first database snapshot

• Support for maintenance windows

• Centralized UI for installation and config of all application and

backup components

These new capabilities and enhancements are why it's worth

upgrading to MongoDB 3.2.

Upgrading MongoDB in Outline

• Manual upgrades – lots of steps, but doable• Automatic – Ops Manager / Cloud Manager

Manual Upgrade

Upgrading a cluster manually

• Follow the steps in the fine documentation:– http://docs.mongodb.org/manual/release-notes/3.2-upgrade/– First result if Googling for “MongoDB Upgrade”

• It's critical to note that upgrade order matters:– For replica sets, first upgrade the secondaries, then the primaries.– For a sharded cluster, follow the 7 steps in the documentation.

Automatic

Upgrading a cluster with Automation

• Ops Manager / Cloud Manager is the easiest, fastest, and best way to upgrade a MongoDB deployment– One-click upgrades.– Rolling upgrades, so zero downtime.

Upgrades

• Select Desired Version from Dropdown

As you can see, upgrading the database software itself is pretty easy.

The fine details

• Backward compatibility & your application• Backward compatibility & your operations• Database behavior & performance• Enabling 3.2 Features

Backward Compatibility

Backward Compatibility: App

• Comprehensive documentation about incompatible changes– https://docs.mongodb.org/manual/release-notes/3.2-compatibility

• Highlights– You may need to upgrade your applications' drivers before upgrading

MongoDB!– Some commands/methods deprecated, removed, changed.

• Audit your codebase for uses of deprecated functionality.

Backward Compatibility: Ops

• Comprehensive documentation about incompatible changes– https://docs.mongodb.org/manual/release-notes/3.2-compatibility

• Highlights– Some settings deprecated, others mmapv1-only.– Stricter replica set configuration validation– mongo* tools' options changed– Legacy (pre-2.6) user account model removed

• Audit scripts & other infrastructure you've got around.

Database Behavior & Performance

Database Behavior & Performance

• Always test your application with the new database version before upgrading production– Apps sometimes rely on unspecified behaviors (e.g. stability of results

from unsorted queries)– Query optimizer improvements may affect index selection (check

important queries with explain)

Good practice: Test against Staging

• Run all your performance correctness tests against a staging environment for some time before upgrading production.

• Best practice: run a real application workload against a 3.2 staging environment, too.– Replicate the workload manually– Use Facebook’s Flashback tool

In our experience, upgrading everything around the database software is where the work is.

Recap

Upgrading Revisited

• Plan• Practice• Productionize

Plan Your Upgrade

• Review the compatibility notes– Audit apps, ops for deprecated details

• Upgrade drivers, scripts, if necessary– Might require some recoding, in edge cases

• Make sure you've got good tests– App behavior, performance, etc.

• Write a checklist of upgrade steps for your environments

Practice Your Upgrade

• Upgrade apps & database in staging environment• Test, test, test.

– If no problems appear within (say) a week of continuous testing, probably a decent candidate.

Productionize Your Upgrade

• Upgrade apps & database in production environment– Ideally, during minimum load/traffic conditions, just for good measure

MongoDB's team can help you!

How we help users upgrade

• MongoDB's Professional Services team offers consulting engagements specifically for upgrading:– To ensure upward compatibility for your apps, tools– To assist with performance & behavior testing around database upgrades– To prepare & execute upgrade plans with your team in your environments.

Contact us for more info: http://mongodb.com/upgrade

consulting@mongodb.com