Download pdf - Building a Cloud-based Social Network with Zend Framework and Doctrine 2

© 2011 Mayflower GmbH

Thorsten Rinne I 11th October 2011

Building a Cloud-based Social Network with Zend Framework and Doctrine 2

Building a Cloud-based social network I Mayflower GmbH I 11th October 2011 I

Team LeadScrum MasterHead of Open Source Labs

@ThorstenRinne2


Last year a new customer came up with an idea...

3


A Social Network!

4


A Social Network?

5

for friends.Building a Cloud-based social network I Mayflower GmbH I 11th October 2011 I 6

Building a Cloud-based social network I Mayflower GmbH I 11th October 2011 I 7

for business.


for videos.

for photos.Building a Cloud-based social network I Mayflower GmbH I 11th October 2011 I 9

for real estate?Building a Cloud-based social network I Mayflower GmbH I 11th October 2011 I 10


The idea:Connect people, buildings

and businesses.

11


Business

People

Building


Business

People

Building

... is living at ...


Business

People

Building

... is living at ...... is the home of ...


Business

People

Building

... is an employee of ...... is living at ...... is the home of ...


Business

People

Building

... is an employee of ...... is living at ...... is the home of ...

... has as employee ...


Business

People

Building

... is an employee of ...... is living at ...

... has its office at ...

... is the home of ...



Business

People

Building

... is an employee of ...... is living at ...

... has its office at ...

... is the home of ...


... is the office location of ...


We thought:Nice idea, let‘s start!

13


4 Requirements

14


1Agile software

development with Scrum

15



1 product owner (customer)

16



16

1 scrum master (myself)



16

1 scrum master (myself)1 development team


Scrum

CancelGift wrap

Return

Productbacklog

17


Scrum

CancelGift wrap

Return

Sprint2 weeks

Productbacklog

17


Scrum

CancelGift wrap

Sprint2 weeks

Return

Sprint goal

Productbacklog

17


Scrum

CancelGift wrap

Sprint2 weeks

Return

Sprint goal

Sprint backlog

Productbacklog

17


Scrum

CancelGift wrap

Sprint2 weeks

Return

Sprint goal

Sprint backlog

Potentially shippableproduct increment

Productbacklog

17


Scrum

CancelGift wrap

Sprint2 weeks

Return

Sprint goal

Sprint backlog


Productbacklog

Coupons

17


Scrum

Sprint2 weeks

Return

Sprint goal

Sprint backlog


Productbacklog

Gift wrapCoupons

Cancel

17


Scrum

Sprint2 weeks

Return

Sprint goal

Sprint backlog


Productbacklog

Gift wrapCoupons

Cancel

24 hours

17


2Scalability

18


Classic web architecture

19


Browser


Appserver 1 Appserver 2

Browser


Appserver 1 Appserver 2

DB SlaveDB Master

Browser


We built a simple, scalable n-tier architecture.

21


Frontend NodesBackend Nodes

22


Frontend Nodes don‘t save anything. Never.

23


Backend Nodes can be sharded and/or clustered.

24


The solution?The cloud.

25

http://www.flickr.com/photos/nirak/644337888/








3Simplicity

27


Automate as much as possible

28


No new fancy things like Scala, node.js or MongoDB.

29



4Extensibility

31


Lean StartupWe can kick/create new

features all the time

32


Lean StartupDecoupled modules and

classes

33


Architecture

34


Frontnode 1Varnish nginx

PHP 5.3


PHP 5.3

Backendnode 3RabbitMQ

Apache SolrPHP 5.3

Backendnode 2MySQL Slave

MogileFSMemcached

PHP 5.3

Backendnode 1MySQL Master

MogileFSMemcached

PHP 5.3


Is this scalable?

36



PHP 5.3


PHP 5.3


Apache SolrPHP 5.3


MogileFSMemcached

PHP 5.3


MogileFSMemcached

PHP 5.3



3


3


3


3


3


3


3


3


3


3

Multiple Frontnodes


3

Multiple Frontnodes

Backend Sharding


Infrastructure

41


One Ubuntu package to rule them all.

42


Development VMsJenkins CI ServerSelenium Server

Monitoring ServerTesting Server

43


Only on production:Frontend Nodes 11.04

Backend Nodes 10.04 LTS

44





☁☁ ☁

Mayflower DMZ

Eucalyptus Cloud

amazon AWS





☁☁ ☁

Mayflower DMZ

Eucalyptus Cloud

amazon AWSDeveloper Laptopwith Ubuntu VM

(Git repo + completesoftware stack)





☁☁ ☁

Mayflower DMZ

Eucalyptus Cloud

amazon AWSGitorious ServerJenkins CI Server

Puppet MasterSelenium Server





☁☁ ☁

Mayflower DMZ

Eucalyptus Cloud

amazon AWS

2 Frontnodes3 Backendnodes

Development MonitorStaging Monitor





☁☁ ☁

Mayflower DMZ

Eucalyptus Cloud

amazon AWS2 Frontnodes2 Backendnodes

DrupalLive Monitor





Configuration Management done with





Configuration changes are on every VM in 5 Minutes.

#WIN





Application

52


Do you remember?

53


Frontend 1Varnish nginx

PHP 5.3


PHP 5.3


Apache SolrPHP 5.3


MogileFSMemcached

PHP 5.3


MogileFSMemcached

PHP 5.3



Apache SolrPHP 5.3


MogileFSMemcached

PHP 5.3


MogileFSMemcached

PHP 5.3



Apache SolrPHP 5.3Backendnode 2MySQL Slave

MogileFSMemcached

PHP 5.3


MogileFSMemcached

PHP 5.3



Various transformations such as lowercasing, removing plurals, or stemming to increase relevancy.

57

Apache Solr XML REST API

ProfilesDE

Lucene Core

ProfilesEN

Lucene Core

Profilesi18n

Lucene Core

Autosuggest

Lucene Core


Search queries tokenized with a LL(1) Parser

implemented in PHP.58

Search string: „mapolis“

LL(1) parser

„q=mapolis&sort=name desc&fl=name,id,country“

http://localhost:8983/solr/select/?indent=on&q=video&sort=price%20desc&fl=name,id,price

http://localhost:8983/solr/select/?indent=on&q=video&sort=price%20desc&fl=name,id,price


Indexing is done with asynchronous jobs with

RabbitMQ.59

MySQL ApacheSolr

Profile informationautosuggestions

user generated content



Why do we need a message broker?

61


Example:Editing your profile

62


PHPApplication QueueExchange PHP CLI

Script

Publish Routes Consumes

MySQL ApacheSolr

sync

async


Why RabbitMQ?

64



Apache SolrPHP 5.3


MogileFSMemcached

PHP 5.3


MogileFSMemcached

PHP 5.3



Apache SolrPHP 5.3


MogileFSMemcached

PHP 5.3


MogileFSMemcached

PHP 5.3


MogileFSan open source distributed filesystem

66


Application levelNo single point of failureAutomatic file replicationShared nothing architectureLocal filesystem agnostic

67

Storage Nodes(mogstored)


Trackers(mogilefsd)Trackers

(mogilefsd)


Client(API: Perl)

Trackers(mogilefsd)


Tracker‘s database(MySQL)



Application

Host 1 Host 2

MySQLMaster

MySQLSlave

Replication

R/W


Application

Host 1 Host 2

MySQLMaster

MySQLSlave

Keepalive

Replication

R/WW

R


Some notes on our database architecture.

71


We use MySQL as key/value storage.

72


We implemented all relations in the application.

73


We already prepared the database design for sharding

74


Benefits:Very simple, scalable and

flexible architecture

75


Drawbacks:More complexity in the

PHP code. But that‘s okay.

76



PHP 5.3


PHP 5.3


Apache SolrPHP 5.3


MogileFSMemcached

PHP 5.3


MogileFSMemcached

PHP 5.3



PHP 5.3


PHP 5.3



PHP 5.3


PHP 5.3



Doctrine 2is a data-mapper,not ActiveRecord.

80


We use Doctrine 2 as ORM.And we think it‘s great.

81


ORM with Doctrine 2

82

/** * @Entity * @Table(name="role") * */class Mapolis_Entity_Role{ /** * @Id * @Column(name="id", type="integer") * @generatedValue(strategy="IDENTITY") */ protected $_roleId;

/** * @Column(name="parent_id", type="integer") * @var int */ protected $_parentId;


Working with objects

83

/** * Saves the personal interests of user. * * @param Mapolis_Entity_Person $profile * * @return void */public function savePersonalInterests( Mapolis_Entity_Person $profile){ $this->getManager() ->getRepository(self::ENTITY_PERSON) ->update($profile);}


★ Transactional write-behind (batch writes)★ Very fast for small UnitOfWorks

Working with objects/** * Deletes a building. * * @param Mapolis_Entity_Building $building * * @return boolean */ public function deleteBuilding($building) { $this->getManager()->remove($building); return $this->getManager()->flush(); }


Working with objects/** * Show recent invitations * * @return array of $count recent invitations */public function recent($inviter, $count = 5){ $invitations = $this->getManager() ->getRepository(self::ENTITY_INVITATION) ->findBy( array('_inviterId' => $inviter), array('_modified' => 'DESC'), $count ); return $invitations;}



Why did we choose Zend Framework?

87


60 colleagues using Zend Framework in other projects.

88


We use Zend Framework more like a component

library.

89


We use Doctrine 2 instead of Zend_Db_Table.

90


We built our own Autoloader.

91


What did we use fromZend Framework?

92


Zend_ApplicationZend_Controller

Zend_ViewZend_Validate

Zend_TranslationZend_LocaleZend_Cache

93



PHP 5.3


PHP 5.3 from Ubuntu 11.04

95




HTTP Request

98

mapolis Core

mapolis Drupal

nginx on port 81 nginx on port 81

Varnish HTTP accelerator


HTTP Request

98

mapolis Core

mapolis Drupal



GET mapolis.com 80


HTTP Request

98

mapolis Core

mapolis Drupal



GET mapolis.com 80


HTTP Request

98

mapolis Core

mapolis Drupal



GET mapolis.com 80


HTTP Request

99

mapolis Core

mapolis Drupal




HTTP Request

99

mapolis Core

mapolis Drupal



GET mapolis.com/de/start 80


HTTP Request

99

mapolis Core

mapolis Drupal





HTTP Request

99

mapolis Core

mapolis Drupal





Frontend

100

⊕Building a Cloud-based social network I Mayflower GmbH I 11th October 2011 I 101


JavaScript Framework


Supported browsers

789

(10)

3.67

(latest)

14(latest)

5.05.1

(latest)(iOS)


Development

104


The Team:1 Scrum Master

2 DevOps4 PHP/JS Developers1 Frontend Developer

105


Development Tools

106

Used IDE:Zend Studio, PhpStorm, vi and Textmate

Database change management:Liquibase

CSS helper:SASS / Compass


Development LaptopUbuntu VM withLocal Git repo

Complete software stack

ant <everything>107


Some ant commands

108


Some ant commands

108

$ ant db-development


Some ant commands

108


$ ant sync


Some ant commands

108


$ ant sync

$ ant phpunit


Some ant commands

108


$ ant sync

$ ant phpunit

$ ant jslint


Some ant commands

108


$ ant sync

$ ant phpunit

$ ant jslint

$ ant deb


Some ant commands

108


$ ant sync

$ ant phpunit

$ ant jslint

$ ant deb

$ ant build-vm


Typical commit workflow

109


Typical commit workflow

109

$ vi Foo.php$ ant phpunit$ ant jslint$ git add Foo.php$ git commit -m ”Bar”$ git pull$ git push


<target name="phpunit" depends="db-test" description="Run all tests"> <exec executable="phpunit" failonerror="true"> <arg line="--configuration ${basedir}/tests/phpunit-verbose.xml ${basedir}/tests/AllTests.php" /> </exec></target>

110

ant phpunit


One click deployment111


And how does it look now?

112



Questions?

114

Thank you very much!

© 2011 Mayflower GmbH

Contact Thorsten [email protected]+49 89 242054-31@ThorstenRinne

Mayflower GmbHMannhardtstr. 680538 MünchenGermany

Please rate me @ http://joind.in/talk/view/3884

mailto:[email protected]

mailto:[email protected]

http://joind.in/talk/view/3884

http://joind.in/talk/view/3884