Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability...

Agenda Isolate Services Durability Availability Conclusion

Implementing High Availabilitywith PostgreSQL

Dimitri Fontainedimitri@2ndQuadrant.fr

25 Oct. 2012

Dimitri Fontainedimitri@2ndQuadrant.fr Implementing High Availability

1 AgendawhoamiAvailability, DuralibityArchitectures and Replications

2 Isolate ServicesTrafic growth

3 DurabilityData DurabilityData Availability

4 AvailabilityServices AvailabilitySharding

5 ConclusionPostgreSQL Replication: Looking back, looking forwardQuestions

Dimitri Fontaine

2ndQuadrant FrancePostgreSQL Major Contributor

pgloader, prefix, skytools, debian, . . .CREATE EXTENSION

CREATE EVENT TRIGGER

Bi-Directional ReplicationPartitioning

Dimitri Fontaine

Proven Architectures, implemented lots at Hi-Media

Hi-Media (turnover scale: 200 millions e)

3 different activities to get money from the web

Allopass, HiPay: internet (micro) paymentTelecom ServiceAdvertising

PostgreSQL is the heart of the technical platform

Business needs complianceCapacity to adapt to changes

PostgreSQL: Your data is our job

How to ensure both duralibity and availability of your data?

Usual needs:

ReliabilityStabilityPerformancesGrowth capacity (think commercial success)Continuity and Innovation

Usual needs:

Glossary

Some vocabulary

AvailabilityDuralibity (ACID)ArchitecturesReplications

Glossary

Some vocabulary

Glossary

Some vocabulary

Glossary

Some vocabulary

Availability of services or of data?Duralibity (ACID)ArchitecturesReplications

Glossary

Some vocabulary

Glossary

Some vocabulary

Needs first

Needs evolve, solutions must adapt

Start simpleSome first classic stepsHigh Availability of DataHigh Availability of ServicesRead Only Load BalancingRead Write Load Balancing

Needs first

Let’s start simple

Our projet life cycle

Let’s start with the example of a quite simpleproject released as a web application seeing itsneeds evolve with its success.

Scaling out 101

Services Availability

Front servers are statelessWatch out for max_connectionsDon’t you use persistent connections!pgbouncer

Scaling out 101

Using more than a single server and a connection pool

pgbouncer

pgbouncer is able to reuse client and server side connections.

pgbouncer

pgbouncer is able to reuse client and server side connections.

Getting serious: backups

Backup Strategy is the single most important step towards dataavailability

Nightly pg_dump -Fc

Don’t forget pg_dumpall–globals-only

Data Retention7 days of nightly backups7 weeks of weekly backups12 months of monthly backups30 years of yearly backups?

Nightly pg_dump -Fc

Failover, 101

pg_dump, pg_restore

protection against errors and omissionsbeware of restoring timestill a must have for data durabilitywhat about data availability?

Failover, 101

pg_dump, pg_restore

protection against errors and omissionsbeware of restoring timestill a must have for data durabilitywhat about data availability?

Failover, 201

Using physical backups and Point In Time Recovery

Point In Time Recovery, 8.1warm standby, 8.2Archiving and crash recoveryarchive_command

restore_command

walmgr.py, WAL-E

Failover, 201

restore_command

walmgr.py, WAL-E

Failover, 201

restore_command

walmgr.py, WAL-E

Failover, 201

restore_command

walmgr.py, WAL-E

Warm Standby

Implementing Warm Standby

Application Split

When you have separate backoffice and production requirements

Cross replicationSlony, Londiste, BucardoSpecific processing, batchesOff-line processingStill Transactional processingSkytools comes with PGQ

Application Split

Implementing londiste and PGQ

Queueing with PGQ

Off-line processing is better done with PGQ

Mainly written in PLpgSQL (and C)Client API for pythonand PHP

some work is happening for JavaCooperative Worker (Skytools 3)

PGQ: Stable, Reliable, Easy to monitor

Queueing with PGQ

Disaster Recovery and Service Continuity

PostgreSQL 9.1 offers Synchronous Replication and Hot Standby

Hot Standby(A)Synchronous ReplicationStandby connects with libpq

recovery.conf

Continue ArchivingSwitchabe per transactionGood performance level

recovery.conf

Implementing Hot Standby

Scaling Writes

PL/proxy

Scale-up or Scale-out?Remote Procedure CallShardingDistributed databaseAutonomous TransactionsStored Procedures

Scaling Writes

PL/proxy

Scaling Writes

PL/proxy

Scaling Writes

PL/proxy

Scaling Writes

PL/proxy

PL/Proxy

Installing plproxy

Example (install.sql)=# create extension plproxy;=# create extension hashlib;

PL/Proxy

pl/proxy is the integrated sharding layer. Now you have to write allyour SQL in server side functions.

Example (admin/change_group_status.sql)create or replace function admin.change_group_status(

user_name text, status integer)returns void as $BODY$

CLUSTER ’fl_cluster’;RUN ON hash_string(user_name, ’lookup3le’);

$BODY$;

Distributed Sequences 1/2

Same schema on every node, best to avoid overlapping sequences.

Example (distributing sequence)ALTER SEQUENCE foo_id_seq

INCREMENT BY 16;

MINVALUE 297000000000000000MAXVALUE 297999999999999999;

http://www.fotolog.com/<user>/297000000000017139/

MINVALUE 297000000000000000MAXVALUE 297999999999999999;

http://www.fotolog.com/<user>/297000000000017139/

Scaling Writes

Implementing plproxy

Scaling Writes

Implementing plproxy

Distributed High Availability

Retrospective and Future of PostgreSQL Replication

8.1, PITR8.2, Warm Standby8.3, pg_standby9.0, Hot Standby9.1, Synchronous Replication9.2, Cascading Replication9.3, Bi-Directional Replication

Questions?

Meet with us on the booth, join us in the Hallway Track!

Want to win a blue elephant?http://2012.pgconf.eu/feedback/

Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability...

Documents

WhoamI - Vicente Aguilera Díaz...WhoamI 4.1.3 Obtener datos: Fiona Newlands Email Profesión Nombre del establecimiento visitado el 06/11/2015 Ciudad en la que se encontraba Michael

Best practices for MySQL High Availability March 2017 SCALE15X · whoami • Chief Evangelist (in the CTO ofﬁce), Percona Inc • Founding team of MariaDB Server (2009-2016), previously

PABLO $ WHOAMI - TomatinaCONtomatinacon.com/wp-content/uploads/2015/09/Olfateando-To... · 2015. 9. 1. · pablo $ whoami • ingeniero informÁtico & mÁster seguridad informÁtica

网站重构Whoami ---- 偷米饭

WhoamI #BBOOTN. WhoamI Big Buffer Overflow Over The Network Y bla bla bla…

$whoami kaoritter at Tokyo girl geek dinners 2013 #TGGD

High Availability 2001 Press Briefing. Agenda What we’re announcing today…a new vision for High Availability New HA software products VERITAS Cluster

For your high availability and clustering needs Percona ... · For your high availability and clustering needs Ramesh Sivaraman Krunal Bauskar. Agenda What is Good HA eco-system ?

MongoDB Backup & Recovery Field Guide...“golang” ]} `whoami` Agenda ... Backup Tips Store backups in structured paths -YYYYMMDD_HHMM Also backup the ... Start a Mongos

Integer Overflow Protections in Android Dan Austin ... · Dan Austin (oblivion@google.com) May 2016. Proprietary + Confidential Agenda $ whoami Stagefright Sanitizers Sanitizers in

The Future of PostgreSQL High Availability · The Future of PostgreSQL High Availability Robert Hodges - Continuent, Inc. Simon Riggs - 2ndQuadrant. PG Day EU 2009 - Paris Agenda

Windows Azure VMs VMs and Cloud Services VM Availability Images and Disks Managing VMs Agenda

WHOAMI?...WHOAMI? Oscar Martínez Ruiz de Castilla Chalaco Ingeniero Electrónico Magister en Ciencias de la Computación CISM, C)ISSO OSCP, C|EH, C|HFI, C)PTE, C)PTC, C)SWAE, Tu desarrollador/analista

UNA AVENTURA PELIGROSA ATAQUES A NIVEL DE HYPERVISOR · 2016-05-25 · Channel Marketing Manager Bitdefender SE & LATAM. Agenda Whoami La nube y la virtualización Ataques ... Tipos

Future Availability of Minerals:Sustainable Development & the Research Agenda

Agenda Basic Unix Commands (Chapters 2 & 3) Miscellaneous Commands: whereis, which, whoami, finger, passwd, cal, date Working with Files: cat, more, less

Whoami Powerpoint 3

1 Guide to RAC. 2 Agenda Introduction Availability Scalability Manageability Total Cost of Ownership Conclusion

Future Availability of Minerals: Sustainable Development & the Research Agenda

Whoami conte