Modern web applications infrastructure

MODERN WEB APPLICATIONS INFRASTRUCTURE

PRACTICES, FLOWS AND TOOLS

!

<EPAM> 2

!

!

"

# IGOR_ALPERT@EPAM.COM

HTTP://UA.LINKEDIN.COM/IN/IALPERT

IGOR ALPERT

<EPAM> 3

Application !

!

<EPAM> 4

Application

CSS

JS

HTML

!

!

5

Application

<EPAM>

$

Feature #1

Feature #2

Bug fixing

!

Logs

6

Application

<EPAM>

$ %

!

7

Application

<EPAM>

Logs

Doc

!

$ %

8

Application

$

Logs

Config

Docs

!

%

<EPAM>

Packages

9

Application

$

Config

!

%

<EPAM>

Logs

Docs

10

Application

$

Logs

Docs

Config

Tests

!

Packages

%

<EPAM>

11

Application

$ &

!

%

<EPAM>

12

Application

0.5

$ 0.8

1.0

!

&%

<EPAM>

13

Application

0.5

$ 0.8

1.0

!

Dev

Test

Prod

&%

<EPAM>

14

Application

0.5

$ 0.8

1.0

!

Dev

Test

Prod

APP

DB

SV

&% '

<EPAM>

DB

DB

DB

DB

' '

' '

15

Application

$

!

&% "

<EPAM>

16

Application

$

!

&% " 7Errors

Logs

Metrics

<EPAM>

17

Agenda !

$DEVELOPMENT

WORKFLOW

%SOURCE

MANAGEMENT

&BUILD

PIPELINE

"DEPLOY

PLATFORMS

7MONITORING

METRICS

CONTINUOUS

INTEGRATION

#

<EPAM>

$

DEVELOPMENT

WORKFLOW

<EPAM> 19

DEVELOPMENT WORKFLOW $

DEV$(

Managers)QA Support

%

<EPAM> 20

DEVELOPMENT WORKFLOW $

DEV$(

Managers)QA Support

%

<EPAM> 21

DEVELOPMENT WORKFLOW $

(Managers

)QADEV

$

Support%

<EPAM> 22

DEVELOPMENT WORKFLOW $

(DESIGN

)TESTING

&BUILD, DEPLOY

aBUG FIX

'LAUNCHDEVELOP

$

2-4 WEEKS ITERATION

<EPAM> 23

DEVELOPMENT WORKFLOW

&

(

*

$

%

UP TO 1 DAY ITERATION

$

24

DEVELOPMENT WORKFLOW

<EPAM>

SERVERS

1. NGINX

2. APACHE HTTPd

3. TOMCAT

4. IIS

5. NODEJS

6. EXPRESS

7. …

STYLES

1. TWITTER BOOTSTRAP

2. ZURB FOUNDATION

3. COMPASS

4. SASS

5. LESS

6. STYLUS

7. CUSTOM SOLUTION

FRAMEWORKS

1. ANGULARJS

2. BACKBONE

3. EXTJS

4. SENCHA TOUCH

5. EMBERJS

6. METEOR

7. DERBY

8. CUSTOM SOLUTION

TESTING

1. JASMINE

2. MOCHA

3. BUSTER

4. QUNIT

5. KARMA

6. PROTRACTOR

7. NIGHTWATCH

8. INTERN

9. WEBDRIVER.IO

10.CUSTOM SOLUTION

OS

1. WINDOWS

2. OSX

3. UBUNTU

4. CENTOS

5. FREEBSD

6. COREOS

TOOLS

1. GRUNT

2. GULP

3. NPM

4. BOWER

5. YEOMAN

6. CUSTOM SOLUTION

DATABASES

1. MONGODB

2. COUCHDB

3. RIAK

4. REDIS

5. CASSANDRA

SERVICES

1. AMAZON

2. NODEJITSU

3. HEROKU

4. MONGOLAB

5. FIREBASE

6. JOYENT

7. PUSHER

8. DOCKER

$

SERVER SIDE

1. PHP

2. RUBY

3. PYTHON

4. …

!

<EPAM> 25

DEVELOPMENT WORKFLOW $

VIRTUAL MACHINE

Your toolset will ask for them as a dependency

Works better and faster on Unix based systems

NODEJS, RUBY, PYTHON

Ability to compile native modules — easier to “make from source” then in Windows

Building a Module in Windows

WINDOWS

node-gyp, Python, shell, VS, …

<EPAM> 26

DEPENDENCY MANAGEMENT $

Packager for OSX, uses formulas to install software

Homebrew

Mac OSXbrew install [package]

Package manager for Debian GNU/Linux distribution and its variants

Advanced Packaging Tool

Ubuntuapt-get install [package]

The Yellowdog Updater, Modified (yum) is an open-source command-line package management utility for Linux operating

Yum

CentOS

yum install [package]

Chocolatey NuGet is a Machine Package Manager, somewhat like apt-get, but built with Windows in mind

Chocolatey

Windows

choco install [package]

<EPAM> 27

DEPENDENCY MANAGEMENT $

PRCE

NGINX

OPENSSL

MAKEDEPEND

PKG-CONFIG

GIT

GIT-FLOW

brew install git-flow brew install nginx

<EPAM> 28

DEPENDENCY MANAGEMENT $

NVM

APP1

Node 0.11x

NVM

APP2

Node 0.10.31

<EPAM> 29

DEPENDENCY MANAGEMENT $

Node Version Manager - Simple bash script to manage multiple active node.js versions

NodeJS

NVMhttps://github.com/creationix/nvm

RVM is the Ruby enVironment Manager. It manages Ruby application environments and enables switching between them

Ruby

RVMhttps://github.com/wayneeseguin/rvm

It creates an environment that has its own installation directories, that doesn’t share libraries with other virtualenv environments

Python

virtualenv

https://github.com/pypa/virtualenv

A Node version manager for the windows folks out there. Inspired by n and nodenv

NodeJS

Nodisthttps://github.com/marcelklehr/nodist

<EPAM> 30

DEPENDENCY MANAGEMENT $

NPM BOWER

manager install

"dependencies": { "<name>": "<version>", "<name>": "<folder>", "<name>": "<package>" }, "devDependencies": { "<test-framework-name>": "<version>" }

"dependencies": { "<name>": "<version>", "<name>": "<folder>", "<name>": "<package>" }, "devDependencies": { "<test-framework-name>": "<version>" }

manager install

manager install

<EPAM> 31

DEPENDENCY MANAGEMENT $

NPM

GRUNT

PACKAGE.JSON

LODASH

ASYNC

KARMA

Packages:

npm install

npm install —save

Downloads:

Corporate mirrors

npm init

DEV-DEPS

<EPAM> 32

DEPENDENCY MANAGEMENT $

BOWER

LODASH

BOWER

BACKBONE

JQUERY

UNDERSCORE

BOOTSTRAP

Packages:

bower init

bower install

npm install -g

http://bower.io/

&BUILD

PIPELINE

<EPAM> 34

BUILD PIPELINE &

Source

Concatenate

Uglify

SourceMaps

Test

Karma

Protractor

Mocha

Coverage

Preprocess

LESS

SASS

Compass

Assets

Templates

CSS

HTML processing

Images optimization

Watch

LiveReload

Rebuild

Serve

Custom

ChangeLog

Notifications

console.debug

<EPAM> 35

GRUNT

BUILD PIPELINE !

Packages:

grunt

grunt coffee:app

jshint: { // define the files to lint files: ['gruntfile.js', 'src/**/*.js', 'test/**/*.js'], // configure JSHint options: { // more options here globals: { jQuery: true, console: true, module: true } } }

Gruntfile.js

Gruntfile.coffee

http://gruntjs.com/http://gruntjs.com/plugins

Good for file operations like copy/move/save. Configuration over code. Better for small projects with small amount of files.

FILE BASED

Most popular JS task runner. Almost any possible task is available as plugin. You can perform build-in or custom operations. Sync/Async

TONS OF PLUGINS

1

2

<EPAM> 36

GULP

BUILD PIPELINE !

var gulp = require('gulp'); !gulp.task('default', function() { // place code for your default task here });

gulpfile.js

http://gulpjs.comhttp://gulpjs.com/plugins/

Packages:

gulp

grunt coffee

npm install --save-dev gulp

No temporary files. Code over Configuration. Much more faster then Grunt for file-content processing operations.

STREAM BASED

Only 4 API methods: task, watch, src, dest. Easy to write complex flows.

EASY API

1

2

<EPAM> 37

YEOMAN

BUILD PIPELINE !

http://gulpjs.comhttp://gulpjs.com/plugins/

Generators: ~1100

npm install -g generator-angular

yo angular

npm install -g yo$ yo angular:controller myController $ yo angular:directive myDirective $ yo angular:filter myFilter $ yo angular:service myService

Pre-packed with test suites

KARMA TESTS

Easy scaffolding system to generate application components

SCAFFOLDING

Comes with build-in live-reload and grunt task

SERVER

1

2

3

<EPAM> 38

BUILD PIPELINE

SLUSHhttps://www.npmjs.org/package/slush

BROCCOLIhttps://www.npmjs.org/package/broccoli

http://indigounited.com/automaton/

AUTOMATIONJAMhttps://github.com/caolan/jam

BRUNCHhttp://brunch.io/ https://github.com/nickjanssen/angus

ANGUS

!

!"

SOURCE MANAGEMENT

<EPAM> 40

SOURCE "

GIT

Cheap branching — just a reference to commit. Lots of possible merging strategies and conflict resolving. History rewrite

EASY BRANCHING AND MERGING

You can work alone or with a team. You can work with or without central server. Even without network connection.

DISTRIBUTED

You can clone whole repo/branch or just a few commits. All meta-data is stored in one place ~/.git

FAST

1

2

3"

git init !git add git commit !git push —force !git rebase -i git merge !git tag -a

<EPAM> 41

SOURCE "

GITLAB

You can integrate it with your own infrastructure, users base, private services etc

PRIVATELY HOSTED

Fully manageable and configurable Same capabilities — collaboration, forking, wiki and JIRA

TRUSTED BY ENTERPRISE

Hosted on GitHub. You can contribute and write your own features

OPEN-SOURCE

1

2

3

<EPAM> 42

SOURCE "

GITHUB

You are able to create branches, tags, pull-requests, edit and commit files directly from browser

WEB INTERFACE

Do code-review with your team, leave comments, track issues and maintain WIKI pages

CODE REVIEW AND COLLABORATION

You can fork and contribute to any public repository. Home for open-source projects

FORK, PULL REQUEST

1

2

3

Mirror mirror on the wall, who forks best of them all?

<EPAM> 43

SOURCE "

BITBUCKET

Unlike GitHub this service will allow you to create *unlimited* amount of private repositories.

FREE PRIVATE REPOS

Natively integrates with Atlasssian software stack. Trusted by Enterprise.

ENTERPRISE ORIENTED

Spooning - with Bitbucket is a pair programming process as an opposite to forking

SPOONING vs. FORKING

1

2

3

<EPAM> 44

SOURCE "

BRANCHING MODELS

Centralized

Gitflow

Github

Feature branch

Commit

Commit

Commit

Commit

REBASE SQUASH

FEATURE BRANCH

…

REBASE

FEATURE

FEATURE

FEATURE

FEATURE

UPSTREAM BRANCH

<EPAM> 45

SOURCE "

HOOKS

All Git hooks are ordinary scripts that Git executes when certain events occur in the repository. This makes them very easy to install and configure

PRE-COMMIT

PREPARE-COMMIT-MSG

COMMIT-MSG

POST-COMMIT

POST-CHECKOUT

PRE-REBASE

PRE-RECEIVE

UPDATE

POST-RECEIVE

Client Server

http://githooks.com/

SPELLING ERRORS

ENFORCE RULES

EMAIL / SMS NOTIFICATIONS

DEPLOY TRIGGER

Use Cases

<EPAM> 46

SOURCE "

COMMIT MESSAGES

[FIX] Closes: #1234 [BREAKING] Interface of method has been changed… [COMPLETE] Feature (F16:Raptor) is ready for integration [ENV] Some minor refactoring

MAKE YOUR CONVENTION

ISSUE MANAGEMENT

1

2

3 RELEASE NOTES AND CHANGE LOGS

4 ANNOTATIONS: squash! fixup!

CONTINUOUS

INTEGRATION

#

<EPAM> 48

TECHNICAL DEPT

TESTS WILL BE IN THE NEXT RELEASE

LET’S JUST COPY/PASTE FOR NOW

1

2

3

CODE ENTROPY: “IF I TOUCH THAT CODE EVERYTHING WILL BREAK”

4

TODO/FIXME STATEMENTS

CONTINUOUS INTEGRATION #

Fix this code, the only one who can. Is in another company already.

http://en.wikipedia.org/wiki/Technical_debt

5

DOCS? MY CODE IS STATE OF ART

<EPAM> 49

CODE QUALITY

CODE-STYLE: TABS/SPACES, INDENTATION

STATIC ANALYZE AND COMPLEXITY: BUGS, DUPLICATION, STANDARDS

1

2

3

UNIT-TESTS AND COVERAGE4

DOCS AND COMMENTS

AUTOMATE

CONTINUOUS INTEGRATION #

5

TECH BACKLOG

<EPAM> 50

CODE STYLE

CONTINUOUS INTEGRATION #

- JSHINT + STYLISH - PLATO - CODE PAINTER - EDITORCONFIG - JSCS - ESLINT

- AIRBNB - CROCKFORD - GOOGLE - JQUERY - MDCS (THREE.JS) - WIKIMEDIA - YANDEX

- COFFEELINT - CSS - HTML

Other validations JSCS Presets

1

2

3

4

5

Tools

READABILITY

GOOD NAMES

CLEAR LOGIC

POTENTIAL ISSUES

GUIDELINES

<EPAM> 51

CONTINUOUS INTEGRATION

TESTS

#

LONG (hours)

MEDIUM (minutes)

FAST (seconds)

FEEDBACK TIME

UNIT-TESTS

UI

API

TDD (BDD)

BETTER CODE UNDERSTANDING

1

2

3

RELIABILITY

4 RELEASE FASTER

5

MOTIVATIONFILE SYSTEM

DATABASE

HEADLESS

END-TO-END

SERVICES

SMOKE TESTS

<EPAM> 52

CONTINUOUS INTEGRATION

FIXTURES

#

module.exports.Permission = login: name: 'login' title: 'Login Page' ! home: name: 'home' title: 'Home Page' ! student: name: 'student' title: 'Student Section' ! moderator: name: 'moderator' title: 'Admin Section'

module.exports.Role = guest: name: 'Guest' permissions: [ Permissions.login Permissions.home ] ! user: name: 'User' permissions: [ Permissions.home Permissions.student ] ! admin: name: 'Admin' permissions: [ Permissions.home Permissions.student Permissions.moderator ]

Permission.coffee Role.coffee

db.Role

Clean

Role.coffee

<EPAM> 53

CONTINUOUS INTEGRATION

CODE COVERAGE

#

97%

NOT TESTED AREA OF APP

DEAD CODE DETECTION

1

2

3 TESTING QUALITY

4ACCEPTANCE THRESHOLD 70-90%

5 REPORTS

JSCoverage

Istanbul

COVERALS.IO

Blanket

CODECLIMATE

<EPAM> 54

CONTINUOUS INTEGRATION

DOCUMENTATION GENERATION

#

KEEP THEM UP-TO-DATE

METHODS AND CLASSES

1

2

3 PUBLIC API

4USAGE EXAMPLES

5 (RE) USE YOUR TESTS

HTML MARKDOWN

JSDoc

Output

JSDOC* CROJSDOC APIDOCJS DOXMOXDOCCO CODO

Tools

@param @name @class @method @option

@example @depricated @package @private @api

<EPAM> 55

CONTINUOUS INTEGRATION

.dot files

#

.gitignore

.gitattributes

.jshintrc

.jshintignore

.editorconfig

.jscs.json

.travis.yml

SHAREABLE CONFIGURATION

EASY TO MANAGE

1

2

3

IDE INTEGRATION4

ESSENTIAL PART OF YOUR ENVIRONMENT

5 NOT A TRASH

Invest time learning to configure your machine and automate processes

Project System

.bashrc

.zshrc

.gitconfig

.aliases

.functions

.osx

http://dotfiles.github.io/

<EPAM> 56

CONTINUOUS INTEGRATION

IDE

#

Code Completion and Inspection

Refactoring, Metrics

1

2

3 Test framework integration

4 Debugger, Frameworks support

5 Source Version control Issue trackers integration

CODIO CLOUD9 CODEBOX CODENVY

Online IDE

<EPAM> 57

CI SERVERS

CONTINUOUS INTEGRATION #

BUILDS HISTORY AND VISIBILITY

LAST GREEN/RED STATUS

1

2

3 PARALLEL BUILDS

4 MULTIPLE ENVIRONMENTS

5 ROLLBACK

JENKINS TEAMCITY GRUNT CONCRETE STRIDER CI

TOOLS

And badges: http://shields.io/

<EPAM> 58

HOSTED SERVICES

CONTINUOUS INTEGRATION #

CI AS A SERVICE

INTEGRATED WITH GITHUB AND BITBUCKET

1

2

3 BUILD ON SCHEDULE

4 DEPLOY GREEN

5 REPORTS AND STATUS DASHBOARD

circle.yml .travis.yml

Options

TRAVIS CI CODESHIP CIRCLE CI BAMBOO MAGNUM CI SEMAPHORE GONDOR RULTOR

Services

"DEPLOY

PLATFORMS

<EPAM> 60

DEPLOY PLATFORMS

SERVER

#

CUSTOMIZABLE STACK AND ENVIRONMENT

NEED FOR (DEV) OPS

DATA: COMPANY POLICY

OWN SERVICES TO USE

d ef g

g / NGINX APACHE TOMCAT

Ñ SSH BASH 7 SYSLOG

# FILES TMP CACHE

$ SQL NOSQL KEY:VAL

( SERVICES DAEMONS CRON

W IPTABLESm SENDMAIL

w

1

2

3

4

5 INFRASTRUCTURE

K

<EPAM>

%61

DEPLOY PLATFORMS

PaaS

#

CLI DEPLOYS

CONFIGURATION DASHBOARD

1

2

3BUILDING BLOCKS (DISK SPACE, DB, MONITORING, ADDONS)

4 MONITORING

5 SLA AND NODES MANAGEMENT

NODEJITSU HEROKU CLOUD FOUNDRY JOYENT ENGINE YARD

Services

<EPAM> 62

DEPLOY PLATFORMS

Docker

#

YOUR OWN PAAS

OPEN SOURCE

1

2

3EASY SCALE

4 MANY READY TO USE STACKS

5 EASY TO MIGRATE

DEIS FLYNN TSURU OCTOHOST !

TOOLS

7MONITORING EVENTS

METRICS

<EPAM> 64

MONITORING EVENTS

Logs management

#

FORMAT IS IMPORTANT

AGGREGATION AND ROTATION

1

2

3 SEARCH

4 ALERTS AND NOTIFICATIONS

5 WATCHERS

<EPAM> 65

MONITORING EVENTS

Example

#

2014/10/02 13:58:32 [error] 925#0: *14 connect() failed (111: Connection refused) while…

SIZE

DATE

10Mb

LOGSSept

10Mb

Oct

07

<EPAM> 66

MONITORING EVENTS

Logging services

#

PAPERTAIL LOGENTRIES SUMOLOGICBOUNDARY SENTRY GRAYLOG2

Services

APPDB SRV

SYS

%ELASTIC SEARCH

n

8

!

<EPAM> 67

MONITORING

Errors

#

CONTEXT IS IMPORTANT

GROUPING

REPRODUCE

FAIL GRACEFULLY

HISTORY

… RAYGUN TRACKJS AIRBRAKE QBAKA MUSCULA JSERRLOGTRACEKIT

Services

1

2

3

4

5

<EPAM> 68

MONITORING METRICS

HealthPage

#

SERVICE #1 (UP) Ã

SERVICE #3 (10k users / s) Ã

SERVICE #1 (10k ops / s) Ã

ÂFREE SPACE (~2Gb)

HARDWARE

SERVICES AND API

1

2

3 TRANSPORTS

4 PERFORMANCE

5 UPTIME

&SUMMARY

70

SUMMARY

<EPAM>

&

DEVELOPMENT

WORKFLOW

SOURCE

MANAGEMENT

DEPLOY

PLATFORMS

MONITORING

METRICS

Automate

Tests

Small features

BUILD

PIPELINE

CONTINUOUS

INTEGRATION

' " ! " 7#

71

SUMMARY

<EPAM>

&

DEVELOPMENT

WORKFLOW

SOURCE

MANAGEMENT

DEPLOY

PLATFORMS

MONITORING

METRICS

Automate

Tests

Small features

Code style

Commits convention

12 factor app

BUILD

PIPELINE

CONTINUOUS

INTEGRATION

' " ! " 7#

72

SUMMARY

<EPAM>

&

DEVELOPMENT

WORKFLOW

SOURCE

MANAGEMENT

DEPLOY

PLATFORMS

MONITORING

METRICS

Automate

Tests

Small features

Code style

Commits convention

12 factor app

Build Blocks

Keep it fast

Run cycles

BUILD

PIPELINE

CONTINUOUS

INTEGRATION

' " ! " 7#

73

SUMMARY

<EPAM>

&

DEVELOPMENT

WORKFLOW

SOURCE

MANAGEMENT

DEPLOY

PLATFORMS

MONITORING

METRICS

Automate

Tests

Small features

Code style

Commits convention

12 factor app

Build Blocks

Keep it fast

Run cycles

Fail early

Fast feedback

Step by Step

BUILD

PIPELINE

CONTINUOUS

INTEGRATION

' " ! " 7#

74

SUMMARY

<EPAM>

&

DEVELOPMENT

WORKFLOW

SOURCE

MANAGEMENT

DEPLOY

PLATFORMS

MONITORING

METRICS

Automate

Tests

Small features

Code style

Commits convention

12 factor app

Build Blocks

Keep it fast

Run cycles

Fail early

Fast feedback

Step by Step

BUILD

PIPELINE

CONTINUOUS

INTEGRATION

DevOps

Production alike

…as a Service

' " ! " 7#

75

SUMMARY

<EPAM>

&

DEVELOPMENT

WORKFLOW

SOURCE

MANAGEMENT

DEPLOY

PLATFORMS

MONITORING

METRICS

Automate

Tests

Small features

Code style

Commits convention

12 factor app

Build Blocks

Keep it fast

Run cycles

Fail early

Fast feedback

Step by Step

BUILD

PIPELINE

CONTINUOUS

INTEGRATION

DevOps

Production alike

…as a Service

Metrics

Checklist

' " ! " 7#

Logs rotation

76

SUMMARY

<EPAM>

&

DEVELOPMENT

WORKFLOW

SOURCE

MANAGEMENT

DEPLOY

PLATFORMS

MONITORING

METRICS

Automate

Tests

Small features

Code style

Commits convention

12 factor app

Build Blocks

Keep it fast

Run cycles

Fail early

Fast feedback

Step by Step

BUILD

PIPELINE

CONTINUOUS

INTEGRATION

DevOps

Production alike

…as a Service

Metrics

Checklist

' " ! " 7#

Logs rotation

77<EPAM>

QUESTIONS?

?

THANKS!