83
DrupalCon Dublin 2016 Capgemini - Digital Factory - Nantes

Meetup - retour sur la DrupalCon Dublin 2016

Embed Size (px)

Citation preview

Page 1: Meetup - retour sur la DrupalCon Dublin 2016

DrupalConDublin

2016Capgemini - Digital Factory - Nantes

Page 2: Meetup - retour sur la DrupalCon Dublin 2016

Yann Jajkiewicz@yjajkiewhttps://yann.me

Jérémy Greffin@jeremygreffinGithub : jgreffin

Page 3: Meetup - retour sur la DrupalCon Dublin 2016

DrupalCon Dublin 2016 Introduction

Page 4: Meetup - retour sur la DrupalCon Dublin 2016

3 joursAvec jusqu’à 11 conférences en parallèle

10 667Tasses de café

1800Personnes sur les 5 jours

Page 5: Meetup - retour sur la DrupalCon Dublin 2016
Page 6: Meetup - retour sur la DrupalCon Dublin 2016

Timeline

DevOps Drupal Echange

Page 7: Meetup - retour sur la DrupalCon Dublin 2016

DevOps Docker

From socketwenchhttps://goo.gl/RFJ6Un

Page 8: Meetup - retour sur la DrupalCon Dublin 2016

Why Docker ?

› Containers› Sandboxes everything› Instructions aren’t enough

Docker vs Vagrant :› Vagrant = multiple VM› Docker = 1 VM for multiple containers

Page 9: Meetup - retour sur la DrupalCon Dublin 2016

Physical hardware

Host OS

Hyperviseur

VM OS VM OS VM OS VM OS

Server A

Server B

Server C

Server D

Vagrant :

Page 10: Meetup - retour sur la DrupalCon Dublin 2016

Physical hardware

Host OS

Container A

Docker :

Docker utilities

Linux Kernel

Docker engine

Container B

Container C

Hyperviseur

Page 11: Meetup - retour sur la DrupalCon Dublin 2016

How to Docker ?

Docker files› Construction d’une image› Description de l’environnement en YAML› 1 fichier par processus

Docker compose› Automatisation du déploiement des images› Définition de l’architecture› 1 commande pour tout installer

Page 12: Meetup - retour sur la DrupalCon Dublin 2016

Docker & Drupal

Go shopping...› Docker4Drupal

› Redis› Memcached› MySQL, MariaDB› D7 & D8› . . .

… or build your own

Page 13: Meetup - retour sur la DrupalCon Dublin 2016

DevOps Kubernetes

From tpryanhttps://goo.gl/3k23PK

Page 14: Meetup - retour sur la DrupalCon Dublin 2016

ProblèmeGestion complexe des VMs et des containers

Kubernetes

SolutionKubernetes gère “magically” différents containers au sein de différentes VMs

Page 15: Meetup - retour sur la DrupalCon Dublin 2016

Kubernetes : késako

› Open Source› Started by Google› Système d’orchestration de containers

Paradigme :› Build scripts VS Desired state› Children VS Employees

Page 16: Meetup - retour sur la DrupalCon Dublin 2016

Kubernetes : paradigme

Employee“We had a tough day, go home and get some sleep”

ChildGo upstairsGet undressedPut on pajamasBrush your teeth

Page 17: Meetup - retour sur la DrupalCon Dublin 2016

Kubernetes : how it works

› Docker files› Pods (1 or + containers), YAML style› Pods controller :

› Réplication, déploiement, service› Gestion des environnements (prod, test)› Autoscaling› Logging, monitoring, web interfacing, cluster

Page 18: Meetup - retour sur la DrupalCon Dublin 2016

Dockercompose

Docker Machine

Docker

Kubernetes

Multiple containers on

same localhost

Cluster of container hosts

replication orchestration scheduling

Manage Remote

Container Hosts

Launch Container hosts in several clouds

Page 19: Meetup - retour sur la DrupalCon Dublin 2016

Everything at Google runs on Containers

› Gmail, Web Search, Maps,...› MapReduce, batch,...› GFS, Colossus,...› Google Platform (VMs run on containers

underneath everything)

Page 20: Meetup - retour sur la DrupalCon Dublin 2016

DevOps Jenkins 2 Pipeline From aroq & shumushin

https://goo.gl/gahNsm

Page 21: Meetup - retour sur la DrupalCon Dublin 2016

Why Jenkins ?› Release process :

› Frequent› Automated› Repeatable› Lowrisk

› Build, deploy, test, release› Continuous delivery› Continuous deployment

Page 22: Meetup - retour sur la DrupalCon Dublin 2016

Jenkins : Pipeline› Plugin› Jenkins 1 Workflow → Jenkins 2 Pipeline› Can survive restarts› Pausable (for human input)› BlueOcean Pugin → better GUI› Jenkinsfile : container for the pipeline› Multibranch pipelines : automatic

pipeline creation for branches

Page 23: Meetup - retour sur la DrupalCon Dublin 2016

Jenkins : pratiques Drupal

› Ne pas commit “vendor”› Pas de composer install ou update en PROD

Solutions :› Build, tarball, sync (doesn’t work with Acquia)› Target repo : commit > CI > Build > commit >

deploy

Page 24: Meetup - retour sur la DrupalCon Dublin 2016

Jenkins : workflow

Git commit & push

Jenkins Pipeline

Gitlab webhookInit Config

BuildStage prepareStage deployStage testing

Prod prepare Prod deploy Prod testing Finalize

Page 25: Meetup - retour sur la DrupalCon Dublin 2016

Jenkins : tools

Page 26: Meetup - retour sur la DrupalCon Dublin 2016

Jenkins : tools

› Docman - one or more source repos + docman config > docman > target repo

› Druflow - deploy drupal-procedures › Drush - especially drush aliases› Behat - testing framework› Pipeline - to orchestrate › Pipeline libraries - make it DRY

Page 27: Meetup - retour sur la DrupalCon Dublin 2016

Drupal Performances

Page 28: Meetup - retour sur la DrupalCon Dublin 2016

Redis or Memcache

Web Server

Web Server

Web Server

Web Server

Bottleneck

Traditional caching implementation

By David Strauss

Page 29: Meetup - retour sur la DrupalCon Dublin 2016

LCache

It combines:› A well established architectural patterns

from modern CPU architecture

From platform.shhttps://goo.gl/1mdkVT

Page 30: Meetup - retour sur la DrupalCon Dublin 2016

Core 0

Existing solutions: Multi-Core Processors

CPU

Core 1

CPU

L1 cache L1 cache

L2 cache

Page 31: Meetup - retour sur la DrupalCon Dublin 2016

LCache

It combines:› A well established architectural patterns

from modern CPU architecture› Efficient data propagation from mysql’s

row-level replication

From platform.shhttps://goo.gl/1mdkVT

› Experience on Valhalla to provide an efficient website object caching

Page 32: Meetup - retour sur la DrupalCon Dublin 2016

Better› Frequently read› Rarely written› Large

LCache

Worse› Read once or not at all› Things handleable

earlier in the stack› Keys updated often› Clearing 100+ keys with

a tag (because of replication)

From platform.shhttps://goo.gl/1mdkVT

Page 33: Meetup - retour sur la DrupalCon Dublin 2016

D8 : practical caching

3 méthodes de cache :› Cache anonyme› Reverse proxy caching› Cache authentifié

From platform.shhttps://goo.gl/1mdkVT

Page 34: Meetup - retour sur la DrupalCon Dublin 2016

D8 : practical caching

› Cache anonyme:› Module : Internal page cache› Activé par défaut dans D8› Page cache middleware› Default request policy : no CLI, no session› Default response policy : no kill switch, no

“no_cache” route, no server errorsFrom platform.shhttps://goo.gl/1mdkVT

Page 35: Meetup - retour sur la DrupalCon Dublin 2016

› Cache anonyme :

› Kill switch\Drupal::service(‘ page_cache_kill_switch ’)->trigger();

› No_cache routesystem.cron:

path: ‘cron/{key}’

defaults:

_controller: ‘\Drupal\system\CronController::run’

options:

no_cache: TRUEFrom platform.shhttps://goo.gl/1mdkVT

Page 36: Meetup - retour sur la DrupalCon Dublin 2016

› Reverse proxy caching :

› Varnish, CDN, ...› Important : définir le “page maximum age”› Définir le reverse proxy dans settings.php

$settings[‘reverse_proxy’] = TRUE;

$settings[‘reverse_proxy_addresses’] = array(‘127.0.0.1’, …);

$settings[‘reverse_proxy_header’] = ‘X_CLUSTER_CLIENT_IP;

From platform.shhttps://goo.gl/1mdkVT

Page 37: Meetup - retour sur la DrupalCon Dublin 2016

› Cache authentifié :

› Username block depends on user logged in› Admin links depends on permissions› Help block depends on URL

› Solution : cache contexts› contexts => [‘user’]› contexts => [‘user.permissions’]› contexts => [‘url’]

From platform.shhttps://goo.gl/1mdkVT

Page 38: Meetup - retour sur la DrupalCon Dublin 2016

› Cache authentifié :

› How to invalidate ?› Invalidate dependencies ?

› Solution : cache tags› tags => [‘node_list’]› tags => [‘user:1]› tags => [‘node:1’]

From platform.shhttps://goo.gl/1mdkVT

Page 39: Meetup - retour sur la DrupalCon Dublin 2016

› Cache authentifié :› How to decide to cache ?

From platform.shhttps://goo.gl/1mdkVT

Page 40: Meetup - retour sur la DrupalCon Dublin 2016

› Cache authentifié :› Exemple :

› Metadata :

From platform.shhttps://goo.gl/1mdkVT

Page 41: Meetup - retour sur la DrupalCon Dublin 2016

D8 : practical caching

› Cacheability debugging :› Module renderviz : permet de voir les

metadata dans l’inspecteur du navigateur

From platform.shhttps://goo.gl/1mdkVT

Page 42: Meetup - retour sur la DrupalCon Dublin 2016

Drupal ElasticSearch

Page 43: Meetup - retour sur la DrupalCon Dublin 2016

Drupal & search

› Core module for search

› Last significant update on Drupal 4.7

Page 44: Meetup - retour sur la DrupalCon Dublin 2016

Search API› Abstraction of SolR› Configuration› Hard to customize

Page 45: Meetup - retour sur la DrupalCon Dublin 2016

Search

› Fast› Text analysis› Facets

Exemples : Wikipedia, Amazon, Github, ...

Page 46: Meetup - retour sur la DrupalCon Dublin 2016
Page 47: Meetup - retour sur la DrupalCon Dublin 2016

› Open source project› Developer-Friendly RESTful API

Page 48: Meetup - retour sur la DrupalCon Dublin 2016

How to use :- Get it- Unzip it- Run it

Page 49: Meetup - retour sur la DrupalCon Dublin 2016

› 1 exemple : Save requests -> Percolator

Index content VS search query

Page 50: Meetup - retour sur la DrupalCon Dublin 2016

› How to start ?

2 ways :➔ Combine with search API➔ Custom connectors with . . .

◆ Friendly API◆ Custom code

Page 51: Meetup - retour sur la DrupalCon Dublin 2016

https://www.drupal.org/project/elasticsearch_helper

Page 52: Meetup - retour sur la DrupalCon Dublin 2016

Drupal Bonnes pratiques

Page 53: Meetup - retour sur la DrupalCon Dublin 2016

Configuration management

› Export/import configuration› Configuration installer : installation profile, UI› Override local configuration : settings.php› Drush can ignore configurations about modules› Drush_cmi_tools : use a YAML list of ignored

configurations› Configuration split : blacklist, ignore set, entities

From pescetti, ademarco, bircherhttps://goo.gl/MSjK3T

Page 54: Meetup - retour sur la DrupalCon Dublin 2016

Configuration management

› Use Git as parallel development› Careful about merge, workflow› Safe : export, commit, merge, import, push

› Features for D8 : › analyzes and automatically packages up site

configuration into a set of features› Only for development purposes From pescetti, ademarco, bircher

https://goo.gl/MSjK3T

Page 55: Meetup - retour sur la DrupalCon Dublin 2016

Configuration management

› Client messing with production configuration› Lock configuration on production (settings.php)› Export from prod and merge to dev› Configuration split : choose what to keep

› Deploy content› Use migrate on events› Default_content module : any module

provides hal+json for entities› Deploy module : stage and preview content

From pescetti, ademarco, bircherhttps://goo.gl/MSjK3T

Page 56: Meetup - retour sur la DrupalCon Dublin 2016

Configuration management : modules

› Config_installer : install site from config› Config_readonly : lock conf via UI› Config_update : diff between original & active› Config_devel : helps developing config› Config_split : import/export filtered config› Config_tools : automatically commit config

changes to git› Features : bundle config for re-use› Config_sync : safely import config from

updated modules From pescetti, ademarco, bircherhttps://goo.gl/MSjK3T

Page 57: Meetup - retour sur la DrupalCon Dublin 2016

Drupal Migration

Page 58: Meetup - retour sur la DrupalCon Dublin 2016

Migrate with D8

1. Requirements2. Anatomy of a migration 3. Migration Framework 4. Performance tips

Page 59: Meetup - retour sur la DrupalCon Dublin 2016

1. REQUIREMENTS

Page 60: Meetup - retour sur la DrupalCon Dublin 2016

1. Migrate is in core !

Tree new modules› Migrate

Handle migration. Framework.› Migrate Drupal

Contains migration from D6 & D7› Migrate Drupal UI

The older migrate UI (new in 8.1.x)

Page 61: Meetup - retour sur la DrupalCon Dublin 2016

1. Migrate is in core !

How to execute migrations ?

› UI ?› Drush command ?

Contrib to Migrate tools & Migrate Plus

Page 62: Meetup - retour sur la DrupalCon Dublin 2016

2. ANATOMY OF A MIGRATION

Page 63: Meetup - retour sur la DrupalCon Dublin 2016

2. Workflow of a migration

DestinationProcess

Process

Process

Source

Page 64: Meetup - retour sur la DrupalCon Dublin 2016

DEFINITIONSYaml Files. Custom files

2. In files

PLUGINSPHP Files Core or custom files Types:

› Source › Process › Destination › Builder › ID Map

Page 65: Meetup - retour sur la DrupalCon Dublin 2016

An easy example

Page 66: Meetup - retour sur la DrupalCon Dublin 2016

DEFINITION config/install/migrate_plus.migration.article_node.yml

Page 67: Meetup - retour sur la DrupalCon Dublin 2016

PLUGINSsrc/Plugin/migrate/source/ArticleNode.php

Page 68: Meetup - retour sur la DrupalCon Dublin 2016

EXECUTIONOnly with Drush8 (8.1-dev) and migrate_tools enabled

Page 69: Meetup - retour sur la DrupalCon Dublin 2016

3. PLUGINS

Page 70: Meetup - retour sur la DrupalCon Dublin 2016

3. SourceHere we tell SqlBase: - Which Database is the Source

Page 71: Meetup - retour sur la DrupalCon Dublin 2016

3. Source- And which Plugin will make the Query

Page 72: Meetup - retour sur la DrupalCon Dublin 2016

3. DestinationHow and where to store the data

- entity:<place-here-an-entity>

Need more destination plugins? ⇒ Search for “destination:” in core

Page 73: Meetup - retour sur la DrupalCon Dublin 2016

3. ID MAPPINGHow Migrate associates old rows with new rows

Page 74: Meetup - retour sur la DrupalCon Dublin 2016

3. PROCESSHow we transform each field, each file or data.

You are able to: › Map fields: Same value as origin. › Modify: Change or process the value. › Add: Create new fields from other fields or calculate

these fields.

Page 75: Meetup - retour sur la DrupalCon Dublin 2016

3. PROCESSMap fields. Values are equal in both sides

Page 76: Meetup - retour sur la DrupalCon Dublin 2016

3. PROCESSDefaultValue. Add a default value

Page 77: Meetup - retour sur la DrupalCon Dublin 2016

3. PROCESSCallable. Values are related but a process is needed with a function

Page 78: Meetup - retour sur la DrupalCon Dublin 2016

3. PERFORMANCE TIPS

Page 79: Meetup - retour sur la DrupalCon Dublin 2016

4. Performance tips

1. Hardware improvements (ssd, cpu,..)2. Disabling functionalities (hook, modules)3. Incremental migration (hash)4. Multithreaded migrations

Page 80: Meetup - retour sur la DrupalCon Dublin 2016

4. Performance tips

$ drush mi cm_articles_1_400 && \ drush mi cm_articles_401_800 &&

Page 81: Meetup - retour sur la DrupalCon Dublin 2016

D8 : Bonnes pratiques en vrac

› Drush is 10x faster than composer› Librairie “composer-patches” : appliquer des

patchs via composer› Pre-commit hook : PHP Lint check› PHPCBF : PHP Code Beautifier & Formatter› Kint : debug in twig› Smacks: organize CSS & JS› Templating : set var in twig (no preprocess)

Bonus D7 : drush unsuck

Page 82: Meetup - retour sur la DrupalCon Dublin 2016

MERCI !Questions ?

Page 83: Meetup - retour sur la DrupalCon Dublin 2016