Embed Size (px)
Citation preview
HOW CAN I BACKUP MY
POSTGRESQL DATABASES?
CUSTOM SCRIPTS, WAL_G, PG_BACKREST,
PG_PROBACKUP, BARMAN??
WHICH ONE SHOULD I USE?
IDENTIFY YOUR NEEDS TO FIND THE RIGHT TOOL FOR YOU!
BARMAN IN ACTION!Giulio Calacoci Senior Developer @ 2ndQuadrant
DataOps 2019 Barcellona
2ndquadrant.com
@asdmaster @2ndQuad #PostgreSQL #DataOps #Barcellona #BarmanInAction
ABOUT MYSELF
▸ Open Source passionate since early 2k
▸ Member of the Italian and European PostgreSQL community
▸ Lean and DevOps practitioner
▸ Open Source Developer
▸ Member of the Barman team
▸ Continuous Delivery Architect @2ndQuadrant
▸ 24/7 support engineer @2ndQuadrant
2ndquadrant.com
@asdmaster @2ndQuad #PostgreSQL #DataOps #Barcellona #BarmanInAction
OBJECTIVES
• Recovery Point Objective (RPO)
• How much data can I afford to lose?
• Recovery Time Objective (RTO)
• How long will it take me to recover?
2ndquadrant.com
@asdmaster @2ndQuad #PostgreSQL #DataOps #Barcellona #BarmanInAction
INCREMENTAL APPROACH
• Keep your goals in mind
• Start (and focus) from disaster recovery
• Incrementally build a DR solution for PostgreSQL (based on barman)
• Evolve from that
2ndquadrant.com
@asdmaster @2ndQuad #PostgreSQL #DataOps #Barcellona #BarmanInAction
PLAYGROUND USED IN THIS TALK
https://github.com/2ndquadrant-it/ansible-postgresql-barman-playground
4 machines Vagrant playground environment with Ansible Playbooks for provisioning of PostgreSQL and Barman
Available under GNU GPL 3
After the talk explore the playbook and “play” with it
LET’S START
STEP 1: CREATING THE CLUSTER
2ndquadrant.com
@asdmaster @2ndQuad #PostgreSQL #DataOps #Barcellona #BarmanInAction
THE “BEATLES” CLUSTER
4 CentOS pre-configured virtual machines:
• John: PostgreSQL 11 + barman-cli on 192.168.33.11 •Paul: PostgreSQL 11 + barman-cli on
192.168.33.10 •George: Barman 2.8 on
192.168.33.12 •Ringo: Barman 2.8 on 192.168.33.13
2ndquadrant.com
@asdmaster @2ndQuad #PostgreSQL #DataOps #Barcellona #BarmanInAction
WHAT TO EXPECT FROM THE PLAYGROUND
• Basic PostgreSQL configuration
• PGDATA initialised with data checksums
• Archive mode enabled
• Syslog destination for logs
• Enable md5 access from 192.168.33.x in pg_hba • Empty database for pgbench
• Paul and John have a running PostgreSQL instance
• SSH key exchange between postgres and barman users
STEP 2: VERIFY THE SETUP
2ndquadrant.com
@asdmaster @2ndQuad #PostgreSQL #DataOps #Barcellona #BarmanInAction
GOALS
• Verify that PostgreSQL is up and running on Paul
• Check PostgreSQL configuration
• Generate some data
Paul
PostgreSQL Master
STEP 3: PREPARE POSTGRESQL
2ndquadrant.com
@asdmaster @2ndQuad #PostgreSQL #DataOps #Barcellona #BarmanInAction
GOALS
• Prepare PostgreSQL for interacting with barman: • Create 'barman' superuser on Paul • Create 'streaming_barman' replication user on
Paul
STEP 4: OUR FIRST BACKUP
2ndquadrant.com
@asdmaster @2ndQuad #PostgreSQL #DataOps #Barcellona #BarmanInAction
GOALS
• Setting up barman • Basic barman concepts • Configuring barman for backing up Paul • Configuring barman for using streaming archive
• First backup using barman
2ndquadrant.com
@asdmaster @2ndQuad #PostgreSQL #DataOps #Barcellona #BarmanInAction
ARCHITECTURE
Continuous backup
streaming wal
STEP 5: INCREMENTAL BACKUP
2ndquadrant.com
@asdmaster @2ndQuad #PostgreSQL #DataOps #Barcellona #BarmanInAction
GOALS
• Setting up barman to be more space-efficient: • Incremental backups concepts • Incremental backups options • copy • link
STEP 6: REMOTE RECOVERY
2ndquadrant.com
@asdmaster @2ndQuad #PostgreSQL #DataOps #Barcellona #BarmanInAction
STARTING ARCHITECTURE
barman recover
Paul
George
John
2ndquadrant.com
@asdmaster @2ndQuad #PostgreSQL #DataOps #Barcellona #BarmanInAction
GOALS
• Remote recovery on John
• Setup of John’s new cluster as Paul’s standby
• Setting on-demand WAL fetching
2ndquadrant.com
@asdmaster @2ndQuad #PostgreSQL #DataOps #Barcellona #BarmanInAction
FINAL ARCHITECTURE
barman_restore_walPaul
George
John
STEP 7: WAL STREAMING
2ndquadrant.com
@asdmaster @2ndQuad #PostgreSQL #DataOps #Barcellona #BarmanInAction
GOALS
• Connect John’s cluster to the Paul’s one
• Setting up WAL streaming
Paul
George
John
WAL streaming
Paul barman_restore_wal
STEP 8: ZERO DATA LOSS
2ndquadrant.com
@asdmaster @2ndQuad #PostgreSQL #DataOps #Barcellona #BarmanInAction
GOALS
• Achieving zero data loss (RPO = 0) using the architecture we built so far
• synchronous standby from master and barman
2ndquadrant.com
@asdmaster @2ndQuad #PostgreSQL #DataOps #Barcellona #BarmanInAction
ARCHITECTURE
barman_restore_wal
barman recover
Synchronous
ZERO DATA LOSS
STEP 9: GEO-REDUNDANCY
2ndquadrant.com
@asdmaster @2ndQuad #PostgreSQL #DataOps #Barcellona #BarmanInAction
GOALS
• Setting up geo-redundancy using Ringo as external barman passive server
• Backing up your backups!
2ndquadrant.com
@asdmaster @2ndQuad #PostgreSQL #DataOps #Barcellona #BarmanInAction
ARCHITECTURE
Paul
George
John
WAL streaming
Ringo
barman sync
Data centre 1 (EU) Data centre 2 (USA)
BUT… THERE IS MORE!
2ndquadrant.com
@asdmaster @2ndQuad #PostgreSQL #DataOps #Barcellona #BarmanInAction
ARCHITECTURE EVOLUTION (1)
Paul
George
John
WAL streaming
Ringo
barman sync
Data centre 1 (EU) Data centre 2 (USA)
Bert
barman recover
2ndquadrant.com
@asdmaster @2ndQuad #PostgreSQL #DataOps #Barcellona #BarmanInAction
ARCHITECTURE EVOLUTION (2)
Paul
George Ringo
barman sync
Data centre 1 (EU) Data centre 2 (USA)
John
barman backupbarman backup
OTHER BARMAN FEATURES
2ndquadrant.com
@asdmaster @2ndQuad #PostgreSQL #DataOps #Barcellona #BarmanInAction
BARMAN FEATURES
• Backup frequency (cron for “barman backup”)
• Retention policies
• Redundancy (number of backups)
• Point of Recoverability (based on time)
• Archive command (barman-wal-archive)
• Streaming backup
• Point In Time Recovery (define a target)
•
BARMAN IN THE FUTURE!
2ndquadrant.com
@asdmaster @2ndQuad #PostgreSQL #DataOps #Barcellona #BarmanInAction
BARMAN FUTURES
• Direct cloud storage (starting with AWS S3) - 2.9
• barman-cloud-wal-archive • barman-cloud-wal-restore • barman-cloud-backup • barman-cloud-recover
• Integration of the above with Barman - 2.10/3.0?
•Multi-tier setups, based on retention policies:
• Tier 2: Local compressed backups
• Tier 3: Cloud storage
THANK YOU!
2ndquadrant.com
@asdmaster @2ndQuad #PostgreSQL #DataOps #Barcellona #BarmanInAction
VIDEOS!
• https://asciinema.org/a/CO270x2NNfBi9vZPS6orM5zRE• https://asciinema.org/a/rz3edZO7RoaqHOrZ2M3oid85x• https://asciinema.org/a/cNdxNdIZQE5xOf1vBVhbM27ft• https://asciinema.org/a/KXRk5NE8DcXaScElXTfXji7vT• https://asciinema.org/a/fuC4xNVGbyfHQZH9UgEShhzYu• https://asciinema.org/a/qEFfCDpPxxzz6cAktbBb92otZ• https://asciinema.org/a/0NEYGKrwUeqtxF94OneWdOR2t• https://asciinema.org/a/2QGItjUKHhYfA70vX3FDIMxfb• https://asciinema.org/a/eG5168n1xLPpFApI9UvvP4jSZ
2ndquadrant.com
@asdmaster @2ndQuad #PostgreSQL #DataOps #Barcellona #BarmanInAction
SPECIAL THANKS TO:
• The Beatles!
• The missing members Barman development team
• For Ansible automation:
• Rubens Souza
• Anna Bellandi
• Benjamin Zander for Rule #6:
• https://www.youtube.com/watch?v=M-HG6X6fpBM
• You for surviving/not falling asleep
2ndquadrant.com
@asdmaster @2ndQuad #PostgreSQL #DataOps #Barcellona #BarmanInAction
LICENCE
Attribution 4.0 International (CC BY 4.0)
You are free to:
• Share — copy and redistribute the material in any medium or format
• Adapt — remix, transform, and build upon the material for any purpose, even commercially.
The licensor cannot revoke these freedoms as long as you follow the license terms.
