View
4
Download
0
Category
Preview:
Citation preview
Introduction What are backups for Backup types Backups considerations Recovering Finally Question
You forgot to put the WHERE in theDELETE?
Martín Marqués
2ndQuadrant
October 18, 2019
Martín Marqués You forgot to put the WHERE in the DELETE?
Introduction What are backups for Backup types Backups considerations Recovering Finally Question
Content
1 Introduction
2 What are backups for
3 Backup types
4 Backups considerations
5 Recovering
6 Finally
7 Question
Martín Marqués You forgot to put the WHERE in the DELETE?
Introduction What are backups for Backup types Backups considerations Recovering Finally Question
What are we going to talk about
What backups are forDifferent ways to backup postgres databasesDisaster scenariosRecovery optionsFinal notes
Martín Marqués You forgot to put the WHERE in the DELETE?
Introduction What are backups for Backup types Backups considerations Recovering Finally Question
Content
1 Introduction
2 What are backups for
3 Backup types
4 Backups considerations
5 Recovering
6 Finally
7 Question
Martín Marqués You forgot to put the WHERE in the DELETE?
Introduction What are backups for Backup types Backups considerations Recovering Finally Question
What are backups for?
Recover from disastersSpin-up replicasCreate development environmentsArchive can serve as a WAL hub
Martín Marqués You forgot to put the WHERE in the DELETE?
Introduction What are backups for Backup types Backups considerations Recovering Finally Question
Types of disasters to recover from
Server irrecoverableTable goneData goneCorruption
Martín Marqués You forgot to put the WHERE in the DELETE?
Introduction What are backups for Backup types Backups considerations Recovering Finally Question
Types of disasters to recover from
Server irrecoverable → DC catches fire/gets floodedTable gone → Devops drops/truncates a tableData gone → Yesterdays deploy has a bugCorruption → HW corrupts data
Martín Marqués You forgot to put the WHERE in the DELETE?
Introduction What are backups for Backup types Backups considerations Recovering Finally Question
Content
1 Introduction
2 What are backups for
3 Backup types
4 Backups considerations
5 Recovering
6 Finally
7 Question
Martín Marqués You forgot to put the WHERE in the DELETE?
Introduction What are backups for Backup types Backups considerations Recovering Finally Question
Backup types
Logical backupspg_dumppglogicalin-core logical replication
Physical backupsBasebackup + WAL files
External OS apps: rsync, tar, etc.pg_basebackupSpecific postgres tools
Standby: sync, async, delayed
Martín Marqués You forgot to put the WHERE in the DELETE?
Introduction What are backups for Backup types Backups considerations Recovering Finally Question
Backup types
Logical backupspg_dumppglogicalin-core logical replication
Physical backupsBasebackup + WAL files
External OS apps: rsync, tar, etc.pg_basebackupSpecific postgres tools
Standby: sync, async, delayed
Martín Marqués You forgot to put the WHERE in the DELETE?
Introduction What are backups for Backup types Backups considerations Recovering Finally Question
Logical Backups
Use pg_dump for taking backupsUse pg_restore to restore the backup
Martín Marqués You forgot to put the WHERE in the DELETE?
Introduction What are backups for Backup types Backups considerations Recovering Finally Question
Logical Backups
ProsExtremely robustBackups are forward compatiblePartial recovery is an optionFlexible backup and recovery options
ConsHolds a snapshot for the whole time pg_dump runsRecovers to snapshot taken, no PITR
Martín Marqués You forgot to put the WHERE in the DELETE?
Introduction What are backups for Backup types Backups considerations Recovering Finally Question
Logical Backups
ProsExtremely robustBackups are forward compatiblePartial recovery is an optionFlexible backup and recovery options
ConsHolds a snapshot for the whole time pg_dump runsRecovers to snapshot taken, no PITR
Martín Marqués You forgot to put the WHERE in the DELETE?
Introduction What are backups for Backup types Backups considerations Recovering Finally Question
Logical Backups
pg_dumpall -g | psql -h remote_host -p 5433SETSETSETCREATE ROLEALTER ROLECREATE ROLEALTER ROLEERROR: role "postgres" already existsALTER ROLE
Martín Marqués You forgot to put the WHERE in the DELETE?
Introduction What are backups for Backup types Backups considerations Recovering Finally Question
Logical Backups
$ pg_dump -Fc martin -f martin.dmp -Z 7
$ ls -l martin.dmp-rw-r--r--. 1 postgres postgres 5207257 oct 17 11:14 martin.dmp
$ pg_restore -C -d postgres -j 4 -p 5433 martin.dmp
Martín Marqués You forgot to put the WHERE in the DELETE?
Introduction What are backups for Backup types Backups considerations Recovering Finally Question
Physical backups
Using OS/Storage toolsConfigure WAL archivingStarts with pg_start_backup()
Do a filesystem backup with: rsync, tar, storagesnapshot, etc.Ends with pg_stop_backup()
Martín Marqués You forgot to put the WHERE in the DELETE?
Introduction What are backups for Backup types Backups considerations Recovering Finally Question
Physical Backups
Using OS/Storage toolsPros
Doesn’t hold a postgres snapshotPoint in time recoveryIncremental backups are possibleVersatility of compression optionsPossible to do work in parallel
ConsNeed to restore the whole clusterRequires manual scripting or tool
Martín Marqués You forgot to put the WHERE in the DELETE?
Introduction What are backups for Backup types Backups considerations Recovering Finally Question
Physical backups
Using pg_basebackup$ pg_basebackup -h hoth -p 5432 -X stream -R -c fast -P \
-D ~/11/data735619/735619 kB (100%), 1/1 tablespace
Martín Marqués You forgot to put the WHERE in the DELETE?
Introduction What are backups for Backup types Backups considerations Recovering Finally Question
Physical Backups
Using pg_basebackup
ProsDoesn’t hold a postgres snapshotCan do point in time recoveryDoesn’t require operating system access to server
ConsNeed to restore the whole clusterCan’t do incremental backupsBackup work can not be done in parallel (WIP patch)Doesn’t backup configuration files outside PGDATA
Martín Marqués You forgot to put the WHERE in the DELETE?
Introduction What are backups for Backup types Backups considerations Recovering Finally Question
How does it work
+---> Basebackup|+-+ WAL archiving| |=============================+-+
time 1 2 3 4 5 6
Martín Marqués You forgot to put the WHERE in the DELETE?
Introduction What are backups for Backup types Backups considerations Recovering Finally Question
How does it work
Incident <---+|
========================X-------
time 1 2 3 4 5 6
Martín Marqués You forgot to put the WHERE in the DELETE?
Introduction What are backups for Backup types Backups considerations Recovering Finally Question
How does it work
+---> Recover basebackup| +-> Recovery target| |========================o
Replay archived WALs
time 1 2 3 4 5 6
Martín Marqués You forgot to put the WHERE in the DELETE?
Introduction What are backups for Backup types Backups considerations Recovering Finally Question
Standby: sync, async, delayed
Create a replica from a physical backupConfigure primary_conninfo,recovery_min_apply_delay, synchronous_commit andsynchronous_standby_names
Martín Marqués You forgot to put the WHERE in the DELETE?
Introduction What are backups for Backup types Backups considerations Recovering Finally Question
Content
1 Introduction
2 What are backups for
3 Backup types
4 Backups considerations
5 Recovering
6 Finally
7 Question
Martín Marqués You forgot to put the WHERE in the DELETE?
Introduction What are backups for Backup types Backups considerations Recovering Finally Question
Conditions a backup needs
Finish successfully
Located in a remote locationBackup and restore process is documented/automatedA restore was attempted successfully
Martín Marqués You forgot to put the WHERE in the DELETE?
Introduction What are backups for Backup types Backups considerations Recovering Finally Question
Conditions a backup needs
Finish successfullyLocated in a remote location
Backup and restore process is documented/automatedA restore was attempted successfully
Martín Marqués You forgot to put the WHERE in the DELETE?
Introduction What are backups for Backup types Backups considerations Recovering Finally Question
Conditions a backup needs
Finish successfullyLocated in a remote locationBackup and restore process is documented/automated
A restore was attempted successfully
Martín Marqués You forgot to put the WHERE in the DELETE?
Introduction What are backups for Backup types Backups considerations Recovering Finally Question
Conditions a backup needs
Finish successfullyLocated in a remote locationBackup and restore process is documented/automatedA restore was attempted successfully
Martín Marqués You forgot to put the WHERE in the DELETE?
Introduction What are backups for Backup types Backups considerations Recovering Finally Question
Content
1 Introduction
2 What are backups for
3 Backup types
4 Backups considerations
5 Recovering
6 Finally
7 Question
Martín Marqués You forgot to put the WHERE in the DELETE?
Introduction What are backups for Backup types Backups considerations Recovering Finally Question
Types of disasters to recover from
Server irrecoverableTable gone / Data goneCorruption
Martín Marqués You forgot to put the WHERE in the DELETE?
Introduction What are backups for Backup types Backups considerations Recovering Finally Question
Server gone
Server irrecoverable
Martín Marqués You forgot to put the WHERE in the DELETE?
Introduction What are backups for Backup types Backups considerations Recovering Finally Question
Server gone - Option 1
Pick the standby with highest LSN and promote itRe-configure other standbys to connect to new primaryRecover, if possible, the old primary as a new standby
Martín Marqués You forgot to put the WHERE in the DELETE?
Introduction What are backups for Backup types Backups considerations Recovering Finally Question
Server gone - Option 2
Bring up a new serverRecover latest base backupReplay all available archived WALs
Martín Marqués You forgot to put the WHERE in the DELETE?
Introduction What are backups for Backup types Backups considerations Recovering Finally Question
Server gone
Table gone / Data gone
Martín Marqués You forgot to put the WHERE in the DELETE?
Introduction What are backups for Backup types Backups considerations Recovering Finally Question
Table/Data gone - Option 1, delayed standby
Check if the data is still in the delayed standbyIf it is, pause replayExtract the missing data from this standbyResume replayApply/merge recovered data on primary node
Martín Marqués You forgot to put the WHERE in the DELETE?
Introduction What are backups for Backup types Backups considerations Recovering Finally Question
Table/Data gone - Option 2, delayed standby
Check if the data is still in the delayed standbyIf it is, pause replaySet recovery_target_time to a time before the incidentSet recovery_target_action to promoteResume replayProvides less downtimeSimplerThere could be more data loss
Martín Marqués You forgot to put the WHERE in the DELETE?
Introduction What are backups for Backup types Backups considerations Recovering Finally Question
Table/Data gone - Option 3, using logical backup
Extract missing data from a pg_dump backupApply/merge recovered data on primary nodeThere may be data loss
Martín Marqués You forgot to put the WHERE in the DELETE?
Introduction What are backups for Backup types Backups considerations Recovering Finally Question
Table/Data gone - Option 4
Recover a physical base backup on new serverUse recovery_target_time to recover up to before thedisaster
Extract missing data and apply on primary nodePromote the recovered backup as the new primary
Martín Marqués You forgot to put the WHERE in the DELETE?
Introduction What are backups for Backup types Backups considerations Recovering Finally Question
Server gone
Data corruption
Martín Marqués You forgot to put the WHERE in the DELETE?
Introduction What are backups for Backup types Backups considerations Recovering Finally Question
Data Corruption, FS corruption, missing files
Almost always only option is a physical backup recoveryRecover base backupReplay WALs to a consistent state, or an LSN at which themissing files are availableVerify if a good version of the corrupt or missing file ispresentKeep replaying WALs if needed and repeat previous step
Martín Marqués You forgot to put the WHERE in the DELETE?
Introduction What are backups for Backup types Backups considerations Recovering Finally Question
Content
1 Introduction
2 What are backups for
3 Backup types
4 Backups considerations
5 Recovering
6 Finally
7 Question
Martín Marqués You forgot to put the WHERE in the DELETE?
Introduction What are backups for Backup types Backups considerations Recovering Finally Question
Conclusions
Take backups!Use some automated toolStandbys can serve as backupsAlways test your backs by recoveringA delayed standby could provide lower RTO in case ofdisasterYou can achieve RPO zero by streaming WALs instead ofusing archive_command
Security first: encrypt your backups
Martín Marqués You forgot to put the WHERE in the DELETE?
Introduction What are backups for Backup types Backups considerations Recovering Finally Question
Conclusions
Take backups!
Use some automated toolStandbys can serve as backupsAlways test your backs by recoveringA delayed standby could provide lower RTO in case ofdisasterYou can achieve RPO zero by streaming WALs instead ofusing archive_command
Security first: encrypt your backups
Martín Marqués You forgot to put the WHERE in the DELETE?
Introduction What are backups for Backup types Backups considerations Recovering Finally Question
Conclusions
Take backups!Use some automated tool
Standbys can serve as backupsAlways test your backs by recoveringA delayed standby could provide lower RTO in case ofdisasterYou can achieve RPO zero by streaming WALs instead ofusing archive_command
Security first: encrypt your backups
Martín Marqués You forgot to put the WHERE in the DELETE?
Introduction What are backups for Backup types Backups considerations Recovering Finally Question
Conclusions
Take backups!Use some automated toolStandbys can serve as backups
Always test your backs by recoveringA delayed standby could provide lower RTO in case ofdisasterYou can achieve RPO zero by streaming WALs instead ofusing archive_command
Security first: encrypt your backups
Martín Marqués You forgot to put the WHERE in the DELETE?
Introduction What are backups for Backup types Backups considerations Recovering Finally Question
Conclusions
Take backups!Use some automated toolStandbys can serve as backupsAlways test your backs by recovering
A delayed standby could provide lower RTO in case ofdisasterYou can achieve RPO zero by streaming WALs instead ofusing archive_command
Security first: encrypt your backups
Martín Marqués You forgot to put the WHERE in the DELETE?
Introduction What are backups for Backup types Backups considerations Recovering Finally Question
Conclusions
Take backups!Use some automated toolStandbys can serve as backupsAlways test your backs by recoveringA delayed standby could provide lower RTO in case ofdisaster
You can achieve RPO zero by streaming WALs instead ofusing archive_command
Security first: encrypt your backups
Martín Marqués You forgot to put the WHERE in the DELETE?
Introduction What are backups for Backup types Backups considerations Recovering Finally Question
Conclusions
Take backups!Use some automated toolStandbys can serve as backupsAlways test your backs by recoveringA delayed standby could provide lower RTO in case ofdisasterYou can achieve RPO zero by streaming WALs instead ofusing archive_command
Security first: encrypt your backups
Martín Marqués You forgot to put the WHERE in the DELETE?
Introduction What are backups for Backup types Backups considerations Recovering Finally Question
Conclusions
Take backups!Use some automated toolStandbys can serve as backupsAlways test your backs by recoveringA delayed standby could provide lower RTO in case ofdisasterYou can achieve RPO zero by streaming WALs instead ofusing archive_command
Security first: encrypt your backups
Martín Marqués You forgot to put the WHERE in the DELETE?
Introduction What are backups for Backup types Backups considerations Recovering Finally Question
Content
1 Introduction
2 What are backups for
3 Backup types
4 Backups considerations
5 Recovering
6 Finally
7 Question
Martín Marqués You forgot to put the WHERE in the DELETE?
Introduction What are backups for Backup types Backups considerations Recovering Finally Question
?
Martín Marqués You forgot to put the WHERE in the DELETE?
Recommended