- 1. Optimiser les performances Drupal depuis les
tranchesSaturday, May 26, 12
2. Qui suis-je?Client Advisor Produits : Managed Cloud, Dev
Cloud, Drupal Gardens,Acquia Commons, Dev Desktop... Offres :
conseil, support et expertise Drupal Nos clients : Twitter, Intel,
Ebay, Paypal, Al Jazeera, WorldEconomic Forum, nombreux sites de
gouvernements, institutions,organisations, etc.. Tutoriels vidos
Drupal 7+ en franais 500+ visites par jour / 1000+ abonns 600+
abonns YouTube / 200k+ vueswww.drupalfacile.org
@DrupalFacileSaturday, May 26, 12 3. Quelques faits Un problme de
cache est presque toujours la cause dun lancement rat Seul un load
test rigoureux vous dira avec exactitude quel trafc vous pouvez
absorber Il y a virtuellement beaucoup dargent perdre pour avoir
voulu en conomiser un peu auparavant Les (trs) grands comptes ne
font pas ncessairement mieux que les "petits" Mieux vaut une quipe
humble qui suit les bonnes pratiques Drupal que dexcellents
dveloppeurs PHP qui rinventent la roueSaturday, May 26, 12 4.
Causes principales dune pointe de traffic ? Lancement de site
Evnement planifi (webinar, sortie dun nouveau produit, fait divers,
soldes, vnement annuel...) Buzz innatendu (rseau social, mdias,
polmique...) Attaque massive (DDoS, DoS...) ...Saturday, May 26, 12
5. 0 1M+ pages vues en 3 heuresSaturday, May 26, 12 6. Anatomie
dune attaque DDoS* Trafc illgitime *Distributed Denial of
ServiceSaturday, May 26, 12 7. Comment faire face unepointe de
traffic ?Saturday, May 26, 12 8. Effectuez un test de monte en
charge ! ABSaturday, May 26, 12 9. Gestion du traffic anonyme et
authentifinginx (serveur HTTP et reverse proxy)Varnish (reverse
proxy cache)memcached (mmoire cache distribue) Stack LAMP (Linux,
Apache, MySQL, PHP) (+ APC)Saturday, May 26, 12 10. Une mine dinfo
: les headers HTTPQuels outils utiliser ?Firebug pour FirefoxWebkit
Inspector Expiration du cache : 1hOu via cURL... $ curl -s -D
/dev/stderr http://site.com Varnish HITSSaturday, May 26, 12 11. Le
problme avec Drupal 6...Solution :Cookie de session Supprime le
cookie de session de D6 Gre le cache externe (Varnish) Mise en
cache des alias dURL Cookie cache bypass (forms, CAPTCHAs...)
Backport de SimpleTest Rplication de base de donnes ... Varnish
MISS...ou Drupal 7 !Saturday, May 26, 12 12. Checklist des caches
Cache de page, cache de blocs, aggrgation CSS et JavaScript, cache
des modules (Views,Panels, Date...)Pour que Pressow fonctionne avec
un reverse proxy cache choisissez le cache externe Jusqu Drupal
7.4, pour faire fonctionner Varnish vous devez ajouter la ligne
suivante dans settings.php : $conf[page_cache_invoke_hooks] =
FALSE;Monitorez vos HITS Varnish avec Firebug, Webkit Inspector our
cURLMonitorez les get_hits et get_misses memcache avec la commande
: $ watch "(echo stats ; echo quit ) | nc SERVER_ID 11211" NE
PURGEZ PAS LES CACHES (Drupal, Varnish) AUX HEURES DE POINTE
!Saturday, May 26, 12 13. Acquia Insight - Quel score auriez-vous ?
Recommandations diverses Performances Scurit Bonnes pratiques
Drupal SEO Grader (partenariat Volacci) Analyse des donnes
Examination de la conguration Analyse du code (hacks,
updates...)Saturday, May 26, 12 14. Le tuning de votre serveur est
vital ! Rservez 60% de la RAM pour MySQLMonitorez la sant de vos
serveurs Prfrez Percona MySQL$ iostat -mx 1 Utilisez PHP FastCGI
(mod_fcgi) $ dstat -lcm 5 Limitez le pool Apache ~60 procs$ mytop
-d mysql$ varnishtop -i TxHeader Surveillez les logs (Apache,
MySQL...) La recette du (7500*.6)+(128x20) = 7060sysadmin ?
(TotalRAM*.AllocMySQL)+(memory_limit*PHP procs) = RAMutilSaturday,
May 26, 12 15. Dbugger grce la ligne de commande... Vrication
rapide de modules dsactiver en prod : $ drush pml --type=module
--status=enabled | egrep (backup_migrate|
boost|dblog|devel|diff|masquerade|migrate|performance|statistics|_ui)
Compter le nombre de pages 404 renvoyes par Apache : $ grep ""
4[0-9][0-9] " access.log | wc -l Obtenir un rapport du nombre de
rponses HTTP 503 (Page Temporary Unavailable) par URL : $ awk { if
($9 == 503) print $0 } access.log | awk {print $7} | sort | uniq -c
| sort -rn | head -n 20 Compter et classer le nombre de HITS Apache
par site : $ wc -l /var/log/sites/*/logs/SERVEUR/access.log | sort
-n Dterminer limpact des crawlers sur votre trafc : $ grep "bot"
access.log | awk {print $14} | sort | uniq -c | sort -rn | head -n
20Saturday, May 26, 12 16. Les bonnes pratiques en place, quen
est-il de la base de donnes ?Saturday, May 26, 12 17. Ceci est une
requte SQL... SELECT DISTINCT "user" AS content_type, u.name AS
content_title, u.uid AS content_author, u.created AS content_date,
u.picture AS content_teaser, u.uid AS content_id,
(points.points*100000+101) AS content_score, 0 AS score FROM users
u INNER JOIN (select * from prole_values where (value LIKE %ICT%)
)pv1 ON u.uid = pv1.uid INNER JOIN (select * from prole_values
where (value LIKE %for%) )pv3 ON u.uid = pv3.uid INNER JOIN (select
* from prole_values where (value LIKE %lifelong%) )pv5 ON u.uid =
pv5.uid INNER JOIN (select * from prole_values where (value LIKE
%learning%) )pv7 ON u.uid = pv7.uid LEFT JOIN userpoints points ON
points.uid=u.uid UNION ALL SELECT "comment" AS content_type,
c.subject AS content_title, c.uid AS content_author, c.jdoestamp AS
content_date, c.comment AS content_teaser, c.cid AS content_id,
t.voting content_score, 0 AS score FROM comments c left JOIN
(SELECT content_id, AVG(value) as voting FROM votingapi_vote where
content_type = comment GROUP BY content_id) as t on c.cid =
t.content_id left join term_comment tc on tc.cid = c.cid left join
term_data td on tc.tid = td.tid where ( (td.name like %ICT% or
c.subject like %ICT%) AND (td.name like %for% or c.subject like
%for%) AND (td.name likeTemps dexcution moyen ? %lifelong% or
c.subject like %lifelong%) AND (td.name like %learning% or
c.subject like %learning%)) GROUP BY c.cid UNION ALL SELECT n.type
AS content_type, n.title AS content_title, n.uid AS content_author,
n.changed AS content_date, nr.teaser AS content_teaser, i.sid AS
content_id, AVG(votes.value) AS content_score, 5 * (1.0E-6 *
SUM(i.score * t.count)) + 5 * POW(2, (GREATEST(MAX(n.created),
MAX(n.changed),10+ secondes... MAX(c.last_comment_jdoestamp)) -
1323851948) * 6.43e- + 5 * (2.0 - 2.0 / (1.0 + MAX(c.comment_count)
* 0.019230769230769)) + 5 * (2.0 - 2.0 / (1.0 + MAX(nc.totalcount)
* 8.7926001477157E-6)) AS score FROM search_index i INNER JOIN
search_total t ON i.word = t.word INNER JOIN node n ON n.nid =
i.sid INNER JOIN search_dataset d ON i.sid = d.sid AND i.type =
d.type LEFT JOIN node_comment_statistics c ON c.nid = i.sid LEFT
JOIN node_counter nc ON nc.nid = i.sid INNER JOIN node_revisions nr
ON nr.nid=n.nid INNER JOIN votingapi_vote votes ON
votes.content_id=n.nid ANDvotes.content_type=node WHEREn.status=1
AND (n.type NOT IN
(award,award_category,award_jury_assignment,award_jury_criterion,award_vote,banner_type,book,cases_awards_submission_type,cases_d
ocumentation_type,cases_awards_type,cases_contact_type,cases_related_type,event,event_type,journal_xed_type,mailout_template,newsle
tter,newsletter_edition,newsletter_section,page,poll,professional_data,resources_type,webform,wiki,workshop_type,cases_reference_type,e
banner,phpcode_type,workshop_highlighted_cases,workshop_participate,workshop_presentation_and_docs,workshop_registration_settings,wor
kshop_speaker) ) AND (i.word LIKE ict% OR i.word LIKE for% OR
i.word LIKE lifelong% OR i.word LIKE learning%) AND i.type = node
AND (d.data LIKE ict% AND d.data LIKE for% AND d.data LIKE
lifelong% AND d.data LIKE learning%) GROUP BY i.type, i.sid HAVING
COUNT(*) >= 4 ORDER BY content_date DESC LIMIT 0, 10Saturday,
May 26, 12 18. Morceau choisi... SELECT DISTINCT "user" AS
content_type, u.name AS content_title, u.uid AS content_author,
u.created AS content_date, u.picture AS content_teaser, u.uid AS
content_id, (points.points*100000+101) AS content_score, 0 AS score
FROM users u INNER JOIN (select * from prole_values where (value
LIKE %ICT%) )pv1 ON u.uid = pv1.uid INNER JOIN (select * from
prole_values where (value LIKE %for%) )pv3 ON SELECT n.type AS
content_type, u.uid = pv3.uid INNER JOIN (select * from
prole_values where (value LIKE %lifelong%) )pv5 ON u.uid = pv5.uid
INNER JOIN (select * from prole_values where (value AS
content_title, n.title LIKE %learning%) )pv7 ON u.uid = pv7.uid
LEFT JOIN userpoints points ON points.uid=u.uid UNION ALL SELECT
"comment" AS content_type, c.subject AS content_title, c.uid AS
content_author, c.jdoestamp AS content_date, c.comment AS
content_teaser, c.cid n.uid AS content_author, AS content_id,
t.voting content_score, 0 AS score FROM comments c left JOIN
(SELECT content_id, AVG(value) as voting FROM votingapi_vote
n.changed AS content_date, where content_type = comment GROUP BY
content_id) as t on c.cid = t.content_id left join term_comment tc
on tc.cid = c.cid left join term_data td on tc.tid = td.tid where
nr.teaser AS content_teaser, like %ICT%) AND (td.name like %for% or
c.subject like %for%) AND (td.name like( (td.name like %ICT% or
c.subject %lifelong% or c.subject like %lifelong%) AND (td.name
like %learning% or c.subject like %learning%)) GROUP BY c.cid UNION
ALL SELECT i.sid ASAS content_title, n.uid AS content_author,
n.changed AS content_date, nr.teaser AS content_teaser, i.sid AS
content_id, n.type AS content_type, n.titlecontent_id,
AVG(votes.value) AS content_score, 5 * AS content_score, *
t.count)) + 5 SUM(i.score * t.count)) + 5 * AVG(votes.value)
(1.0E-6 * SUM(i.score 5 * (1.0E-6 * * POW(2,
(GREATEST(MAX(n.created), MAX(n.changed),
MAX(c.last_comment_jdoestamp)) - 1323851948) * 6.43e- + 5 * (2.0 -
2.0 / (1.0 + MAX(c.comment_count) * 0.019230769230769)) + 5 * (2.0
- 2.0 / POW(2, (GREATEST(MAX(n.created), (1.0 + MAX(nc.totalcount)
* 8.7926001477157E-6)) AS score FROM search_index i INNER JOIN
search_total t ON i.word = t.word INNER JOIN node MAX(n.changed), n
ON n.nid = i.sid INNER JOIN search_dataset d ON i.sid = d.sid AND
i.type = d.type LEFT JOIN node_comment_statistics c ON c.nid =
i.sid LEFT JOIN node_counter nc ON nc.nid = i.sid INNER JOIN
node_revisions nr ON nr.nid=n.nid INNER JOIN votingapi_vote2.0 /
(1.0 + MAX(c.last_comment_jdoestamp)) - 1323851948) * 6.43e- + 5 *
(2.0 - votes ON votes.content_id=n.nid ANDv o t e s
.MAX(c.comment_count) * 0.019230769230769)) + 5=* (2.0 - 2.0 /
Dcontent_type=node WHEREn.status 1 A N (1.0 (+ n.typeNOT IN
(award,award_category,award_jury_assignment,award_jury_criterion,award_vote,banner_type,book,cases_awards_submission_type,cases_d
MAX(nc.totalcount) * 8.7926001477157E-6)) AS score
ocumentation_type,cases_awards_type,cases_contact_type,cases_related_type,event,event_type,journal_xed_type,mailout_template,newsle
FROM search_index i;
tter,newsletter_edition,newsletter_section,page,poll,professional_data,resources_type,webform,wiki,workshop_type,cases_reference_type,e
banner,phpcode_type,workshop_highlighted_cases,workshop_participate,workshop_presentation_and_docs,workshop_registration_settings,wor
kshop_speaker) ) AND (i.word LIKE ict% OR i.word LIKE for% OR
i.word LIKE lifelong% OR i.word LIKE learning%) AND i.type = node
AND (d.data LIKE ict% AND d.data LIKE for% AND d.data LIKE
lifelong% AND d.data LIKE learning%) GROUP BY i.type, i.sid HAVING
COUNT(*) >= 4 ORDER BY content_date DESC LIMIT 0, 10Saturday,
May 26, 12 19. Requtes MySQL lentes - Rapport New Relic6+ secondes
!Insertions WATCHDOGen base de donnesSaturday, May 26, 12 20. Temps
de chargement dune page 10 secondes pour charger la page dont 7
rien que pourles insertions WATCHDOGSaturday, May 26, 12 21.
Maatkit / Percona Toolkit1. Statistiques de la requte # Query 1:
0.03 QPS, 0.08x concurrency, ID 0x70761915D5D15769 at byte1429910
1# This item is included in the report because it matches --limit.#
Attributepct totalmin max avg 95% stddev median# ========= ======
======= ======= ======= ======= ======= ======= =======# Count
2425482. Nombre de lignes examines # Exec jdoe# Lock jdoe# Rows
sent2814 0 7131s96ms 7.46k 2s0311s4ms3 3s 37us3 5s03958ms368us0
2s03# Rows exam 41 4.83G1.94M 1.94M 1.94M 1.86M0.03 1.86M# Users 1
johndoe3. Temps dexcution moyen# Hosts# Databases29 ded-662.pr...
(565/22%)... 8 more1 johndoe# Jdoe range 2012-03-06 06:33:47 to
2012-03-07 06:47:05# bytes2 179.16k 72727272 072# Rows affe00 0 0 0
0 0 0# Rows read0 7.46k3 3 3 3 0 34. Requte SQL# 1us# 10us 3#
Query_jdoe distribution# 100us# 1ms# 10ms# 100ms#4 1s
################################################################ $
mk-query-digest mysqld-slow.log > slow.txt # 10s+ ##
Tablesou#SHOW TABLE STATUS FROM `johndoe ` LIKE url_aliasG#SHOW
CREATE TABLE `johndoe `.`url_alias`G# EXPLAIN /*!50100
PARTITIONS*/SELECT SUBSTRING_INDEX(src, /, 1) AS path FROM
url_alias GROUP BY pathG $ pt-query-digest mysqld-slow.log >
slow.txtSaturday, May 26, 12 22. Checklist de base de donnes Cf.
checklist des caches - Plus les pages Drupal sont mises en cache,
plus la base dedonnes peut se concentrer sur dautres
tches...Utilisez le moteur InnoDB (row-level locking) plutt que
MyISAM (table-level locking)Suivez les bonnes pratiques MySQL
(index, suppression de rand(), SELECT *...)Utilisez les rapports
New Relic pour savoir quelles requtes SQL doivent tre optimises ou
supprimesSauvegardez vos bases de donnes depuis un hot-spare
MySQLExplorez les alternatives MySQL : MongoDB,
Cassandra...Saturday, May 26, 12 23. Chaque dtail compte. vident ?
Pas tant que a...Saturday, May 26, 12 24. Exemple #1 - Redirections
.htaccess Un chier .htaccess ne devrait jamais contenir plus de 100
redirections Pensez Global Redirect, Path Redirect... Prfrez les
redirections visibles de votre fournisseur DNS plutt que des
RewriteRulesRewriteCond %{HTTP_HOST} ^(www.)?site.com$
[NC]RewriteCond %{REQUEST_URI} ^/quality$ [NC]RewriteRule ^(.*)$
http://newsite.com/services/max-rehab/ 150k+ redirections...1mn
pour ouvrir le chier ! quality [L,R=301] Taille du chier ? 90M+
!RewriteCond %{HTTP_HOST} ^(www.)?site.com$ [NC]RewriteCond
%{REQUEST_URI} ^/satisfaction$ [NC]RewriteRule ^(.*)$
http://newsite.com/services/addons/ Le chier .htaccess nest pas mis
en cache satisfaction [L,R=301]RewriteCond %{HTTP_HOST}
^(www.)?site.com$ [NC] Apache genoux, CPU et I/O saturs...
RewriteCond %{REQUEST_URI} ^/support$ [NC]RewriteRule ^(.*)$
http://newsite.com/customers/products/support [L,R=301]
.htaccessSaturday, May 26, 12 25. Exemple #2 - Utilisation de
LOWER() Drupal 5, 6 et 7 ont tour tour utilis LOWER() car les
oprations LIKE de PostgreSQL sont sensibles la casse et cela
maximisait ainsi la portabilit du code entre SGBD Pressow - qui ne
supporte que MySQL - le premier supprim LOWER() Beaucoup de modules
de contrib continuent malheureusement lutiliser /** * Custom
validation for user login form * * @ingroup logintoboggan_form */
function logintoboggan_user_login_validate($form, &$form_state)
{ if (isset($form_state[values][name]) &&
$form_state[values][name]) { if ($name = db_result(db_query("SELECT
name FROM {users} WHERE LOWER(mail) = LOWER(%s)",
$form_state[values][name]))) { form_set_value($form[name], $name,
$form_state); } } }Saturday, May 26, 12 26. Exemple #3 - Dumps
MySQL # cat /mnt/files/backup/DB.sync.sh #!/usr/bin/env bash
mysqldump -cK --single-transaction PROD675 -u USER -pPASS -h
server.domain | mysql STGdb1 -u USER -pPASS; mysqldump mysqldump
-cK -cK --single-transaction --single-transactionPROD740PROD742-u-u
USER USER-pPASS-pPASS -h -hTaille de chaque base
?server.domainserver.domain||mysqlmysqlSTGdb2 -u USER -pPASS;STGdb3
-u USER -pPASS; mysqldump -cK --single-transaction PROD693 -u USER
-pPASS -h server.domain | mysql STGdb4 -u USER -pPASS; mysqldump
mysqldump -cK -cK --single-transaction
--single-transactionPROD681PROD764-u-u USER USER-pPASS-pPASS -h
-hserver.domainserver.domain2GB minimum...||mysqlmysqlSTGdb5 -u
USER -pPASS;STGdb6 -u USER -pPASS; mysqldump -cK
--single-transaction PROD762 -u USER -pPASS -h server.domain |
mysql STGdb7 -u USER -pPASS; 21 dumps MySQL mysqldump mysqldump -cK
-cK --single-transaction --single-transactionPROD778PROD780-u-u
USER USER-pPASS-pPASS -h
-hserver.domainserver.domain||mysqlmysqlSTGdb8 -u USER
-pPASS;STGdb9 -u USER -pPASS; mysqldump mysqldumpconcurrents -cK
-cK --single-transaction --single-transactionPROD670PROD738-u-u
USER USER-pPASS-pPASS -h
-hserver.domainserver.domain||mysqlmysqlSTGdb17 -u USER
-pPASS;STGdb18 -u USER -pPASS; mysqldump -cK --single-transaction
PROD736 -u USER -pPASS -h server.domain | mysql STGdb19 -u USER
-pPASS; mysqldump -cK --single-transaction PROD679 -u USER -pPASS
-h server.domain | mysql STGdb20 -u USER -pPASS; mysqldump -cK
--single-transaction PROD744 -u USER -pPASS -h server.domain |
mysql STGdb21 -u USER -pPASS; mysqldump mysqldump -cK -cK
--single-transaction --single-transactionPROD746PROD677-u-u USER
USER-pPASS-pPASS -h -hserver.domainserver.domain I/O
saturs||mysqlmysqlSTGdb22 -u USER -pPASS;STGdb23 -u USER -pPASS;
mysqldump mysqldump -cK -cK --single-transaction
--single-transactionPROD768PROD766-u-u USER USER-pPASS-pPASS -h
-hserver.domainserver.domain SWAP permanent||mysqlmysqlSTGdb24 -u
USER -pPASS;STGdb25 -u USER -pPASS;A travers le rseau mysqldump -cK
--single-transaction PROD683 -u USER -pPASS -h server.domain |
mysql STGdb26 -u USER -pPASS; mysqldump mysqldump -cK -cK
--single-transaction --single-transactionPROD961PROD691-u-u USER
USER-pPASS-pPASS -h -hserver.domainserver.domain CPU
lockups||mysqlmysqlSTGdb27 -u USER -pPASS;STGdb28 -u USER -pPASS;
echo The sync of your databases from prod to stage is complete at
`date`| mail -s Database Sync Complete -a "From:[email protected]"
[email protected];Saturday, May 26, 12 27. Exemple #4 - Organisation de
fichiers Larriv du Cloud Computing change les habitudes de stockage
NAS, SAN, GlusterFS ? Le stockage rseau est lent (liste des chiers,
tlchargement...) Adoptez une stratgie dorganisation de chiers
AAAA-MM-JJ Limitez-vous 1000 chiers par rpertoire [15:51:50]
[email protected]:/mnt/gfs/jdoe/files# ls -l | wc -l 204718
[15:53:12] [email protected]:/mnt/gfs/jdoe/files# ls -lh | grep M |
grep pdf | head -n 5 -rw-r--r-- 1 jdoe jdoe2.1M 2011-09-20 22:33
081010_ns_1.pdf -rw-r--r-- 1 jdoe jdoe2.1M 2011-09-20 22:42
081010_ns_1v2.pdf -rw-r--r-- 1 jdoe jdoe1.5M 2011-09-20 15:52
090911_g2_digiblazes.pdf -rwxr-xr-x 1 jdoe jdoe1.6M 2011-09-15
17:07 101112_g5_1.pdf -rw-r--r-- 1 jdoe jdoe2.0M 2011-09-20 20:17
110304_quiz5-6.pdfSELECT ws.uid, f.* FROM file_managed
fCocktailINNER JOIN webform_submitted_data wsd ON f.fid = wsd.data
dtonnantINNER JOIN webform_submissions ws ON ws.sid = wsd.sidWHERE
f.uri = public://files/[FILENAME].pdfGSaturday, May 26, 12 28.
Merci. Questions ?Saturday, May 26, 12 29. Aurlien Navarre
@AurelienNavarre@DrupalFacileSaturday, May 26, 12
