Create a Scalable Mobile Media Site with Drupal and MongoDB

CIGNEX Datamatics Confidential www.cignex.com

CIGNEX Datamatics Confidential

Mobile Media Site using Drupal & MongoDB

Presented By:Yash Badiani, Big Data Practice Lead, CIGNEX DatamaticsGaurav Khambhala, Technical Lead, CIGNEX Datamatics30th July, 2013



About Acquia



Acquia – Enterprise Guide to Drupal

3

Co-founded by the Drupal project’ creator in 2007, Acquia has helped more than 2,000 clients manage

growth and scale of their digital experiences.

Acquia provides wide range of Drupal products and services, including:

Acquia Cloud Enterprise GardensAcquia Network

Drupal Training Professional Services

Drupal Commons



About CIGNEX Datamatics

4



What Does CIGNEX Datamatics Do?

5

Since 2000, CIGNEX Datamatics has implemented over 400 Open Source enterprise

solutions addressing business requirements related to Portals, Content & Big Data



Where We Can Help You

6

SOLUTIONS

Managed Cloud Services - Develop, Deploy, Manage

Annual Product Subscription: Liferay, Alfresco, Magento, Hadoop, Selenium

Extended Development Center – Center of Excellence

UI Development Integration Customization Migration Testing Training Support (24*7)

User eXperience Platform

Portals Liferay, Magento,Drupal, Adobe CQ

• Intranet • Extranet• Social

Collaboration

• Mobile Portals

• E-Commerce

Enterprise Content Management

Content Alfresco, Drupal, Magento, Adobe CQ, Moodle, Ephesoft

• WCM• DM• RM• DAM

• E-Commerce• E-learning• ERP• Imaging Solutions

SERVICES

Making Data WorkBig Data Hadoop, MongoDB,

Hbase, Neo4j, Solr

• Analytics• Mobile• Social• Web• Real-time

• DW - BI

• Log Processing and Analysis

• Enterprise Search

Velocity

Complexity

Volume

Variety



Part of Datamatics (DGSL)

• Mission– Experts in improving

Enterprise productivity through Process Engineering & Information Management Solutions

• Key Highlights– Founded in 1975

– Publicly listed in India

– Annual consolidated revenue of US$100 Million

– Fortune 500 clients

– 4,400+ employees across 22 offices in 9 countries

Strategic Alliances

7



About the presenters

• Yash Badiani is the Big Data Practice Lead at CIGNEX Datamatics and focuses on Big Data Technologies including MongoDB & Hadoop. He has worked extensively on large Data warehousing & Business Intelligence projects with tools such as Business Objects, Microsoft SQL Server, Microstrategy, IBM Cognos.

• Gaurav Khambhala works at CIGNEX Datamatics as Technical Lead. He is

the senior member of the PHP Practice at CIGNEX Datamatics and is involved on various technology initiatives like Big Data where he focuses on integration of PHP with NoSQL sources like MongoDB. He has a wide industry experience in software development & management in Open Source technologies such as Drupal, Moodle & Wordpress.

8



Agenda

• The Mobile Media Use Case• Requirements and Challenges• Solution : Mobile Media site using Drupal & MongoDB• Why Drupal and MongoDB?• Demo and Solution Features• Benefits • Summary

9



The Mobile Explosion!

10

By 2015, at least 60% of information workers will

interact with their content applications via a mobile

device

Employees work on proposals and presentation on

mobile devices while travelling

People use digital assets (videos, images) longer on

Tablets and Mobiles compared to desktops

Based on a report by a leading IT advisory firm



Mobile Media Use Case

• Mobile Media site includes the following features:– Store a variety of Images & associated metadata– Massively Scalable to store billions of images– Access through Mobile– Create /Edit Albums– Add Images to the Albums– Add / Edit Metadata of Images– Search Images / Albums by date, metadata, albums, etc– Social Media features – Likes, comments

11



Requirements of Mobile Media sites

12

• Fast performance• Large user base• Concurrent CRUD• Access through various channels

• Millions of digital assets• Variety of content• Complexity of data

• Rich UI features• Social features• Mobile access• Fast search

• Elastic scaling• Cost effectiveness• Centralized storage• Ease of Maintenance

• HIGH availability• Automatic failover• User management

Velocity Volume

User experience

Scalability

Security & Availability

• Easy integration• Shorter dev cycle• Faster deployment• Ease of schema design

Flexibility & Agility



Standard Three Layered Data Architecture

13

File System

Metadata in RDBMS

Search

Standard Three Layered Storage

Application layer



Limitations of RDBMS

• Support limited to terabytes– No support for petabytes to

exabytes

• Manage only structured data– No support for semi-structured and

unstructured data

• RDBMS don't scale inherently– Scale up/Scale out (Load Balancing

& Replication)

• Hard to shard / partition– Large data files

• Both read / write throughput not possible

– Transactional / Analytical databases

• Specialized hardware - expensive

14

RDBMS can’t manage all dimensions of data with speed & at lower cost.



NoSQL is the right solution

• They are schema less• Designed to support huge data volumes

– Facebook 135 billion messages/month; Twitter 7TB data/day

• Scalable replication and distribution mechanism– Thousands of machines distributed around the world

• Massive write performance with asynchronous inserts and updates• Designed to give high query performance• Runs on commodity hardware• Most NoSQL databases are Open Source

15

Not SQLOnly



NoSQL – Data Models

16

Column FamiliesUsage: Read/Write Intensive

Popular databases: Hbase, Cassandra

Document StoreUsage: Working with Occasionally

changing/consistent dataPopular databases: CouchDB,

MongoDB

Graph Database

Usage: Spatial Data storagePopular databases: Neo4j, Bigdata

Key Value / Tulip StoreUsage: Briskly changing data and

high availabilityPopular databases: Riak, Redis,

Azure Table storage

NoSQL Databases

• 4 broad data models• 120+ variants available in the market



Requirements of Mobile Media sites - Recap

17

• Fast performance• Large user base• Concurrent CRUD• Access through various channels

• Millions of digital assets• Variety of content• Complexity of data

• Rich UI features• Social features• Mobile access• Fast search

• Elastic scaling• Cost effectiveness• Centralized storage• Ease of maintenance

• HIGH availability• Automatic failover• User management

Velocity Volume

User experience

Scalability

Security & Availability

• Easy integration• Shorter dev cycle• Faster deployment• Ease of schema design

Flexibility & Agility

Mobile Media Site



Drupal with MongoDB Solution

18

Themes

CoreModules

Nodes TaxonomyUserRoles

Forms &Menu

CustomModules

Workflow ForumsComments& Ratings

Tagging

PHP

Web Services

3rd party & Internal

Applications

MongoDBDriver

MongosRouting Process

Replica Set

MongoDB MongoDB

Replica Set

MongoDB MongoDB



Why Drupal?

19

PluggableArchitecture

Data Abstraction Layer

Easy to Upgrade

Secure

Active CommunityWidely Adopted

Scalable

3rd Party Tools Integration

User Management& Permissions

HTML5 & CSS Support

Drupal Open Source CMS



Websites using Drupal

20

Website: Whitehouse.gov Website: Data.gov.uk

Website: mtv.co.uk

Website: research.yahoo.com

Website: pdx.eduWebsite:

EndPoverty2015.org



Why MongoDB?

21

Agile and Scalable

Full Index Support

Document Oriented Storage

ReplicationQuerying

Atomic Updates

Data Processing and aggregation

High Availability

Open Source NoSQL Solution



Customers using MongoDB

• Centralized data management platform

• 2 billion+ documents

• 20 TB of photo metadata

• TV episodes and series

• Risk solutions auditing data

22

Source: http://www.10gen.com/customers



Demo

23

• Media site on mobile simulator• Like & comment on an image on mobile simulator• Mobile site on web browser• Verify ‘Like’ & comment of the same image on web browser• Search images & access control



Solution Features

Architecture and Design

24



Architecture

25

User MetadataInd

exes Albums

ImageMetadata GridFS

Form API

Drupal API

CustomModule

Browser/MobileTheme

MongoDB PHP Driver

Menu API

UserMobileDevice

ImageMetadata

GridFS



Add Album

Flow

26

Add Image

View Album

View Individual Images

Like Image

Comment Image

Add Tags to Images

View Counter

Search ImagesBy Tags

User Metadata Albums GridFS Image Metadata

DBRef

DBRef DBRef

DBRef

MongoDB Collections

Use

r A

ctio

ns



27

Schema Design

User Metadata GridFS

Albums Image Metadata

• User ID• DBRef (Album)• Tags• Thumbnail• Likes• View Counter• Comments• Permission

• FS.Files• FS.Chunks

• User ID• Tags• Title

• Make• Model• Date Time• Aperture• Exposure• DBRef (GridFS)



28

Schema DesignImage Metadata

Albums

User Metadata



29

MongoDB Monitoring Service (MMS)

• DB Storage

• Cursors

• Replica Sets

• Network Connections

• Non Mapped Virtual Memory

• Opcounters



Benefits

Drupal MongoDB

Most advanced content management solutions

Scalability – billions of content items, millions of users

Highly customized websites Performance – FAST writes through sharding, reads through indexes

Most search friendly CMS Data safety through replication

Less coding, high on automation Centralized single system for data storage

Powered by 7000+ plugins and extensions

Monitoring through MMS

Active community, real time assistance Enterprise support through 10gen

30



Summary & Key Takeaways

• MongoDB provides the RIGHT fit for CMS applications with flexibility, scale & speed

• Drupal’s advanced & automated CMS features and tight integration with MongoDB makes it the right choice for building agile websites

• Both Drupal & MongoDB are feature rich and being Open Source, provide significant cost benefits

31



Thank you. Questions?

CIGNEX Datamatics makes Open Source work for you!

32

Yash BadianiBig Data Practice Lead

[email protected]

Gaurav KhambhalaTechnical Lead

[email protected]

mailto:[email protected]

mailto:[email protected]

Technology

Create a Scalable Mobile Media Site with Drupal and MongoDB