Meetup - retour sur la DrupalCon Dublin 2016

DrupalConDublin

2016Capgemini - Digital Factory - Nantes

Yann Jajkiewicz@yjajkiewhttps://yann.me

Jérémy Greffin@jeremygreffinGithub : jgreffin

DrupalCon Dublin 2016 Introduction

3 joursAvec jusqu’à 11 conférences en parallèle

10 667Tasses de café

1800Personnes sur les 5 jours

Timeline

DevOps Drupal Echange

DevOps Docker

From socketwenchhttps://goo.gl/RFJ6Un

Why Docker ?

› Containers› Sandboxes everything› Instructions aren’t enough

Docker vs Vagrant :› Vagrant = multiple VM› Docker = 1 VM for multiple containers

Physical hardware

Host OS

Hyperviseur

VM OS VM OS VM OS VM OS

Server A

Server B

Server C

Server D

Vagrant :

Physical hardware

Host OS

Container A

Docker :

Docker utilities

Linux Kernel

Docker engine

Container B

Container C

Hyperviseur

How to Docker ?

Docker files› Construction d’une image› Description de l’environnement en YAML› 1 fichier par processus

Docker compose› Automatisation du déploiement des images› Définition de l’architecture› 1 commande pour tout installer

Docker & Drupal

Go shopping...› Docker4Drupal

› Redis› Memcached› MySQL, MariaDB› D7 & D8› . . .

… or build your own

DevOps Kubernetes

From tpryanhttps://goo.gl/3k23PK

ProblèmeGestion complexe des VMs et des containers

Kubernetes

SolutionKubernetes gère “magically” différents containers au sein de différentes VMs

Kubernetes : késako

› Open Source› Started by Google› Système d’orchestration de containers

Paradigme :› Build scripts VS Desired state› Children VS Employees

Kubernetes : paradigme

Employee“We had a tough day, go home and get some sleep”

ChildGo upstairsGet undressedPut on pajamasBrush your teeth

Kubernetes : how it works

› Docker files› Pods (1 or + containers), YAML style› Pods controller :

› Réplication, déploiement, service› Gestion des environnements (prod, test)› Autoscaling› Logging, monitoring, web interfacing, cluster

Dockercompose

Docker Machine

Docker

Kubernetes

Multiple containers on

same localhost

Cluster of container hosts

replication orchestration scheduling

Manage Remote

Container Hosts

Launch Container hosts in several clouds

Everything at Google runs on Containers

› Gmail, Web Search, Maps,...› MapReduce, batch,...› GFS, Colossus,...› Google Platform (VMs run on containers

underneath everything)

DevOps Jenkins 2 Pipeline From aroq & shumushin

https://goo.gl/gahNsm

Why Jenkins ?› Release process :

› Frequent› Automated› Repeatable› Lowrisk

› Build, deploy, test, release› Continuous delivery› Continuous deployment

Jenkins : Pipeline› Plugin› Jenkins 1 Workflow → Jenkins 2 Pipeline› Can survive restarts› Pausable (for human input)› BlueOcean Pugin → better GUI› Jenkinsfile : container for the pipeline› Multibranch pipelines : automatic

pipeline creation for branches

Jenkins : pratiques Drupal

› Ne pas commit “vendor”› Pas de composer install ou update en PROD

Solutions :› Build, tarball, sync (doesn’t work with Acquia)› Target repo : commit > CI > Build > commit >

deploy

Jenkins : workflow

Git commit & push

Jenkins Pipeline

Gitlab webhookInit Config

BuildStage prepareStage deployStage testing

Prod prepare Prod deploy Prod testing Finalize

Jenkins : tools

Jenkins : tools

› Docman - one or more source repos + docman config > docman > target repo

› Druflow - deploy drupal-procedures › Drush - especially drush aliases› Behat - testing framework› Pipeline - to orchestrate › Pipeline libraries - make it DRY

Drupal Performances

Redis or Memcache

Web Server

Web Server

Web Server

Web Server

Bottleneck

Traditional caching implementation

By David Strauss

LCache

It combines:› A well established architectural patterns

from modern CPU architecture

From platform.shhttps://goo.gl/1mdkVT

Core 0

Existing solutions: Multi-Core Processors

CPU

Core 1

CPU

L1 cache L1 cache

L2 cache

LCache

It combines:› A well established architectural patterns

from modern CPU architecture› Efficient data propagation from mysql’s

row-level replication

From platform.shhttps://goo.gl/1mdkVT

› Experience on Valhalla to provide an efficient website object caching

Better› Frequently read› Rarely written› Large

LCache

Worse› Read once or not at all› Things handleable

earlier in the stack› Keys updated often› Clearing 100+ keys with

a tag (because of replication)

From platform.shhttps://goo.gl/1mdkVT

D8 : practical caching

3 méthodes de cache :› Cache anonyme› Reverse proxy caching› Cache authentifié

From platform.shhttps://goo.gl/1mdkVT

D8 : practical caching

› Cache anonyme:› Module : Internal page cache› Activé par défaut dans D8› Page cache middleware› Default request policy : no CLI, no session› Default response policy : no kill switch, no

“no_cache” route, no server errorsFrom platform.shhttps://goo.gl/1mdkVT

› Cache anonyme :

› Kill switch\Drupal::service(‘ page_cache_kill_switch ’)->trigger();

› No_cache routesystem.cron:

path: ‘cron/{key}’

defaults:

_controller: ‘\Drupal\system\CronController::run’

options:

no_cache: TRUEFrom platform.shhttps://goo.gl/1mdkVT

› Reverse proxy caching :

› Varnish, CDN, ...› Important : définir le “page maximum age”› Définir le reverse proxy dans settings.php

$settings[‘reverse_proxy’] = TRUE;

$settings[‘reverse_proxy_addresses’] = array(‘127.0.0.1’, …);

$settings[‘reverse_proxy_header’] = ‘X_CLUSTER_CLIENT_IP;

From platform.shhttps://goo.gl/1mdkVT

› Cache authentifié :

› Username block depends on user logged in› Admin links depends on permissions› Help block depends on URL

› Solution : cache contexts› contexts => [‘user’]› contexts => [‘user.permissions’]› contexts => [‘url’]

From platform.shhttps://goo.gl/1mdkVT

› Cache authentifié :

› How to invalidate ?› Invalidate dependencies ?

› Solution : cache tags› tags => [‘node_list’]› tags => [‘user:1]› tags => [‘node:1’]

From platform.shhttps://goo.gl/1mdkVT

› Cache authentifié :› How to decide to cache ?

From platform.shhttps://goo.gl/1mdkVT

› Cache authentifié :› Exemple :

› Metadata :

From platform.shhttps://goo.gl/1mdkVT

D8 : practical caching

› Cacheability debugging :› Module renderviz : permet de voir les

metadata dans l’inspecteur du navigateur

From platform.shhttps://goo.gl/1mdkVT

Drupal ElasticSearch

Drupal & search

› Core module for search

› Last significant update on Drupal 4.7

Search API› Abstraction of SolR› Configuration› Hard to customize

Search

› Fast› Text analysis› Facets

Exemples : Wikipedia, Amazon, Github, ...

› Open source project› Developer-Friendly RESTful API

How to use :- Get it- Unzip it- Run it

› 1 exemple : Save requests -> Percolator

Index content VS search query

› How to start ?

2 ways :➔ Combine with search API➔ Custom connectors with . . .

◆ Friendly API◆ Custom code

https://www.drupal.org/project/elasticsearch_helper

Drupal Bonnes pratiques

Configuration management

› Export/import configuration› Configuration installer : installation profile, UI› Override local configuration : settings.php› Drush can ignore configurations about modules› Drush_cmi_tools : use a YAML list of ignored

configurations› Configuration split : blacklist, ignore set, entities

From pescetti, ademarco, bircherhttps://goo.gl/MSjK3T

Configuration management

› Use Git as parallel development› Careful about merge, workflow› Safe : export, commit, merge, import, push

› Features for D8 : › analyzes and automatically packages up site

configuration into a set of features› Only for development purposes From pescetti, ademarco, bircher

https://goo.gl/MSjK3T

Configuration management

› Client messing with production configuration› Lock configuration on production (settings.php)› Export from prod and merge to dev› Configuration split : choose what to keep

› Deploy content› Use migrate on events› Default_content module : any module

provides hal+json for entities› Deploy module : stage and preview content

From pescetti, ademarco, bircherhttps://goo.gl/MSjK3T

Configuration management : modules

› Config_installer : install site from config› Config_readonly : lock conf via UI› Config_update : diff between original & active› Config_devel : helps developing config› Config_split : import/export filtered config› Config_tools : automatically commit config

changes to git› Features : bundle config for re-use› Config_sync : safely import config from

updated modules From pescetti, ademarco, bircherhttps://goo.gl/MSjK3T

Drupal Migration

Migrate with D8

1. Requirements2. Anatomy of a migration 3. Migration Framework 4. Performance tips

1. REQUIREMENTS

1. Migrate is in core !

Tree new modules› Migrate

Handle migration. Framework.› Migrate Drupal

Contains migration from D6 & D7› Migrate Drupal UI

The older migrate UI (new in 8.1.x)

1. Migrate is in core !

How to execute migrations ?

› UI ?› Drush command ?

Contrib to Migrate tools & Migrate Plus

2. ANATOMY OF A MIGRATION

2. Workflow of a migration

DestinationProcess

Process

Process

Source

DEFINITIONSYaml Files. Custom files

2. In files

PLUGINSPHP Files Core or custom files Types:

› Source › Process › Destination › Builder › ID Map

An easy example

DEFINITION config/install/migrate_plus.migration.article_node.yml

PLUGINSsrc/Plugin/migrate/source/ArticleNode.php

EXECUTIONOnly with Drush8 (8.1-dev) and migrate_tools enabled

3. PLUGINS

3. SourceHere we tell SqlBase: - Which Database is the Source

3. Source- And which Plugin will make the Query

3. DestinationHow and where to store the data

- entity:<place-here-an-entity>

Need more destination plugins? ⇒ Search for “destination:” in core

3. ID MAPPINGHow Migrate associates old rows with new rows

3. PROCESSHow we transform each field, each file or data.

You are able to: › Map fields: Same value as origin. › Modify: Change or process the value. › Add: Create new fields from other fields or calculate

these fields.

3. PROCESSMap fields. Values are equal in both sides

3. PROCESSDefaultValue. Add a default value

3. PROCESSCallable. Values are related but a process is needed with a function

3. PERFORMANCE TIPS

4. Performance tips

1. Hardware improvements (ssd, cpu,..)2. Disabling functionalities (hook, modules)3. Incremental migration (hash)4. Multithreaded migrations

4. Performance tips

$ drush mi cm_articles_1_400 && \ drush mi cm_articles_401_800 &&

D8 : Bonnes pratiques en vrac

› Drush is 10x faster than composer› Librairie “composer-patches” : appliquer des

patchs via composer› Pre-commit hook : PHP Lint check› PHPCBF : PHP Code Beautifier & Formatter› Kint : debug in twig› Smacks: organize CSS & JS› Templating : set var in twig (no preprocess)

Bonus D7 : drush unsuck

MERCI !Questions ?