Introduction What are backups for Backup types Backups considerations Recovering Finally Question

You forgot to put the WHERE in theDELETE?

Martín Marqués

2ndQuadrant

October 18, 2019

Martín Marqués You forgot to put the WHERE in the DELETE?

Introduction What are backups for Backup types Backups considerations Recovering Finally Question

Content

1 Introduction

2 What are backups for

3 Backup types

4 Backups considerations

5 Recovering

6 Finally

7 Question

Martín Marqués You forgot to put the WHERE in the DELETE?

Introduction What are backups for Backup types Backups considerations Recovering Finally Question

What are we going to talk about

What backups are forDifferent ways to backup postgres databasesDisaster scenariosRecovery optionsFinal notes

Martín Marqués You forgot to put the WHERE in the DELETE?

Introduction What are backups for Backup types Backups considerations Recovering Finally Question

Content

1 Introduction

2 What are backups for

3 Backup types

4 Backups considerations

5 Recovering

6 Finally

7 Question

Martín Marqués You forgot to put the WHERE in the DELETE?

Introduction What are backups for Backup types Backups considerations Recovering Finally Question

What are backups for?

Recover from disastersSpin-up replicasCreate development environmentsArchive can serve as a WAL hub

Martín Marqués You forgot to put the WHERE in the DELETE?

Introduction What are backups for Backup types Backups considerations Recovering Finally Question

Types of disasters to recover from

Server irrecoverableTable goneData goneCorruption

Martín Marqués You forgot to put the WHERE in the DELETE?

Introduction What are backups for Backup types Backups considerations Recovering Finally Question

Types of disasters to recover from

Server irrecoverable → DC catches fire/gets floodedTable gone → Devops drops/truncates a tableData gone → Yesterdays deploy has a bugCorruption → HW corrupts data

Martín Marqués You forgot to put the WHERE in the DELETE?

Introduction What are backups for Backup types Backups considerations Recovering Finally Question

Content

1 Introduction

2 What are backups for

3 Backup types

4 Backups considerations

5 Recovering

6 Finally

7 Question

Martín Marqués You forgot to put the WHERE in the DELETE?

Introduction What are backups for Backup types Backups considerations Recovering Finally Question

Backup types

Logical backupspg_dumppglogicalin-core logical replication

Physical backupsBasebackup + WAL files

External OS apps: rsync, tar, etc.pg_basebackupSpecific postgres tools

Standby: sync, async, delayed

Martín Marqués You forgot to put the WHERE in the DELETE?

Introduction What are backups for Backup types Backups considerations Recovering Finally Question

Backup types

Logical backupspg_dumppglogicalin-core logical replication

Physical backupsBasebackup + WAL files

External OS apps: rsync, tar, etc.pg_basebackupSpecific postgres tools

Standby: sync, async, delayed

Martín Marqués You forgot to put the WHERE in the DELETE?

Introduction What are backups for Backup types Backups considerations Recovering Finally Question

Logical Backups

Use pg_dump for taking backupsUse pg_restore to restore the backup

Martín Marqués You forgot to put the WHERE in the DELETE?

Introduction What are backups for Backup types Backups considerations Recovering Finally Question

Logical Backups

ProsExtremely robustBackups are forward compatiblePartial recovery is an optionFlexible backup and recovery options

ConsHolds a snapshot for the whole time pg_dump runsRecovers to snapshot taken, no PITR

Martín Marqués You forgot to put the WHERE in the DELETE?

Introduction What are backups for Backup types Backups considerations Recovering Finally Question

Logical Backups

ProsExtremely robustBackups are forward compatiblePartial recovery is an optionFlexible backup and recovery options

ConsHolds a snapshot for the whole time pg_dump runsRecovers to snapshot taken, no PITR

Martín Marqués You forgot to put the WHERE in the DELETE?

Introduction What are backups for Backup types Backups considerations Recovering Finally Question

Logical Backups

pg_dumpall -g | psql -h remote_host -p 5433SETSETSETCREATE ROLEALTER ROLECREATE ROLEALTER ROLEERROR: role "postgres" already existsALTER ROLE

Martín Marqués You forgot to put the WHERE in the DELETE?

Introduction What are backups for Backup types Backups considerations Recovering Finally Question

Logical Backups

$ pg_dump -Fc martin -f martin.dmp -Z 7

$ ls -l martin.dmp-rw-r--r--. 1 postgres postgres 5207257 oct 17 11:14 martin.dmp

$ pg_restore -C -d postgres -j 4 -p 5433 martin.dmp

Martín Marqués You forgot to put the WHERE in the DELETE?

Introduction What are backups for Backup types Backups considerations Recovering Finally Question

Physical backups

Using OS/Storage toolsConfigure WAL archivingStarts with pg_start_backup()

Do a filesystem backup with: rsync, tar, storagesnapshot, etc.Ends with pg_stop_backup()

Martín Marqués You forgot to put the WHERE in the DELETE?

Introduction What are backups for Backup types Backups considerations Recovering Finally Question

Physical Backups

Using OS/Storage toolsPros

Doesn’t hold a postgres snapshotPoint in time recoveryIncremental backups are possibleVersatility of compression optionsPossible to do work in parallel

ConsNeed to restore the whole clusterRequires manual scripting or tool

Martín Marqués You forgot to put the WHERE in the DELETE?

Introduction What are backups for Backup types Backups considerations Recovering Finally Question

Physical backups

Using pg_basebackup$ pg_basebackup -h hoth -p 5432 -X stream -R -c fast -P \

-D ~/11/data735619/735619 kB (100%), 1/1 tablespace

Martín Marqués You forgot to put the WHERE in the DELETE?

Introduction What are backups for Backup types Backups considerations Recovering Finally Question

Physical Backups

Using pg_basebackup

ProsDoesn’t hold a postgres snapshotCan do point in time recoveryDoesn’t require operating system access to server

ConsNeed to restore the whole clusterCan’t do incremental backupsBackup work can not be done in parallel (WIP patch)Doesn’t backup configuration files outside PGDATA

Martín Marqués You forgot to put the WHERE in the DELETE?

Introduction What are backups for Backup types Backups considerations Recovering Finally Question

How does it work

+---> Basebackup|+-+ WAL archiving| |=============================+-+

time 1 2 3 4 5 6

Martín Marqués You forgot to put the WHERE in the DELETE?

Introduction What are backups for Backup types Backups considerations Recovering Finally Question

How does it work

Incident <---+|

========================X-------

time 1 2 3 4 5 6

Martín Marqués You forgot to put the WHERE in the DELETE?

Introduction What are backups for Backup types Backups considerations Recovering Finally Question

How does it work

+---> Recover basebackup| +-> Recovery target| |========================o

Replay archived WALs

time 1 2 3 4 5 6

Martín Marqués You forgot to put the WHERE in the DELETE?

Introduction What are backups for Backup types Backups considerations Recovering Finally Question

Standby: sync, async, delayed

Create a replica from a physical backupConfigure primary_conninfo,recovery_min_apply_delay, synchronous_commit andsynchronous_standby_names

Martín Marqués You forgot to put the WHERE in the DELETE?

Introduction What are backups for Backup types Backups considerations Recovering Finally Question

Content

1 Introduction

2 What are backups for

3 Backup types

4 Backups considerations

5 Recovering

6 Finally

7 Question

Martín Marqués You forgot to put the WHERE in the DELETE?

Introduction What are backups for Backup types Backups considerations Recovering Finally Question

Conditions a backup needs

Finish successfully

Located in a remote locationBackup and restore process is documented/automatedA restore was attempted successfully

Martín Marqués You forgot to put the WHERE in the DELETE?

Introduction What are backups for Backup types Backups considerations Recovering Finally Question

Conditions a backup needs

Finish successfullyLocated in a remote location

Backup and restore process is documented/automatedA restore was attempted successfully

Martín Marqués You forgot to put the WHERE in the DELETE?

Introduction What are backups for Backup types Backups considerations Recovering Finally Question

Conditions a backup needs

Finish successfullyLocated in a remote locationBackup and restore process is documented/automated

A restore was attempted successfully

Martín Marqués You forgot to put the WHERE in the DELETE?

Introduction What are backups for Backup types Backups considerations Recovering Finally Question

Conditions a backup needs

Finish successfullyLocated in a remote locationBackup and restore process is documented/automatedA restore was attempted successfully

Martín Marqués You forgot to put the WHERE in the DELETE?

Introduction What are backups for Backup types Backups considerations Recovering Finally Question

Content

1 Introduction

2 What are backups for

3 Backup types

4 Backups considerations

5 Recovering

6 Finally

7 Question

Martín Marqués You forgot to put the WHERE in the DELETE?

Introduction What are backups for Backup types Backups considerations Recovering Finally Question

Types of disasters to recover from

Server irrecoverableTable gone / Data goneCorruption

Martín Marqués You forgot to put the WHERE in the DELETE?

Introduction What are backups for Backup types Backups considerations Recovering Finally Question

Server gone

Server irrecoverable

Martín Marqués You forgot to put the WHERE in the DELETE?

Introduction What are backups for Backup types Backups considerations Recovering Finally Question

Server gone - Option 1

Pick the standby with highest LSN and promote itRe-configure other standbys to connect to new primaryRecover, if possible, the old primary as a new standby

Martín Marqués You forgot to put the WHERE in the DELETE?

Introduction What are backups for Backup types Backups considerations Recovering Finally Question

Server gone - Option 2

Bring up a new serverRecover latest base backupReplay all available archived WALs

Martín Marqués You forgot to put the WHERE in the DELETE?

Introduction What are backups for Backup types Backups considerations Recovering Finally Question

Server gone

Table gone / Data gone

Martín Marqués You forgot to put the WHERE in the DELETE?

Introduction What are backups for Backup types Backups considerations Recovering Finally Question

Table/Data gone - Option 1, delayed standby

Check if the data is still in the delayed standbyIf it is, pause replayExtract the missing data from this standbyResume replayApply/merge recovered data on primary node

Martín Marqués You forgot to put the WHERE in the DELETE?

Introduction What are backups for Backup types Backups considerations Recovering Finally Question

Table/Data gone - Option 2, delayed standby

Check if the data is still in the delayed standbyIf it is, pause replaySet recovery_target_time to a time before the incidentSet recovery_target_action to promoteResume replayProvides less downtimeSimplerThere could be more data loss

Martín Marqués You forgot to put the WHERE in the DELETE?

Introduction What are backups for Backup types Backups considerations Recovering Finally Question

Table/Data gone - Option 3, using logical backup

Extract missing data from a pg_dump backupApply/merge recovered data on primary nodeThere may be data loss

Martín Marqués You forgot to put the WHERE in the DELETE?

Introduction What are backups for Backup types Backups considerations Recovering Finally Question

Table/Data gone - Option 4

Recover a physical base backup on new serverUse recovery_target_time to recover up to before thedisaster

Extract missing data and apply on primary nodePromote the recovered backup as the new primary

Martín Marqués You forgot to put the WHERE in the DELETE?

Introduction What are backups for Backup types Backups considerations Recovering Finally Question

Server gone

Data corruption

Martín Marqués You forgot to put the WHERE in the DELETE?

Introduction What are backups for Backup types Backups considerations Recovering Finally Question

Data Corruption, FS corruption, missing files

Almost always only option is a physical backup recoveryRecover base backupReplay WALs to a consistent state, or an LSN at which themissing files are availableVerify if a good version of the corrupt or missing file ispresentKeep replaying WALs if needed and repeat previous step

Martín Marqués You forgot to put the WHERE in the DELETE?

Introduction What are backups for Backup types Backups considerations Recovering Finally Question

Content

1 Introduction

2 What are backups for

3 Backup types

4 Backups considerations

5 Recovering

6 Finally

7 Question

Martín Marqués You forgot to put the WHERE in the DELETE?

Introduction What are backups for Backup types Backups considerations Recovering Finally Question

Conclusions

Take backups!Use some automated toolStandbys can serve as backupsAlways test your backs by recoveringA delayed standby could provide lower RTO in case ofdisasterYou can achieve RPO zero by streaming WALs instead ofusing archive_command

Security first: encrypt your backups

Martín Marqués You forgot to put the WHERE in the DELETE?

Introduction What are backups for Backup types Backups considerations Recovering Finally Question

Conclusions

Take backups!

Use some automated toolStandbys can serve as backupsAlways test your backs by recoveringA delayed standby could provide lower RTO in case ofdisasterYou can achieve RPO zero by streaming WALs instead ofusing archive_command

Security first: encrypt your backups

Martín Marqués You forgot to put the WHERE in the DELETE?

Introduction What are backups for Backup types Backups considerations Recovering Finally Question

Conclusions

Take backups!Use some automated tool

Standbys can serve as backupsAlways test your backs by recoveringA delayed standby could provide lower RTO in case ofdisasterYou can achieve RPO zero by streaming WALs instead ofusing archive_command

Security first: encrypt your backups

Martín Marqués You forgot to put the WHERE in the DELETE?

Introduction What are backups for Backup types Backups considerations Recovering Finally Question

Conclusions

Take backups!Use some automated toolStandbys can serve as backups

Always test your backs by recoveringA delayed standby could provide lower RTO in case ofdisasterYou can achieve RPO zero by streaming WALs instead ofusing archive_command

Security first: encrypt your backups

Martín Marqués You forgot to put the WHERE in the DELETE?

Introduction What are backups for Backup types Backups considerations Recovering Finally Question

Conclusions

Take backups!Use some automated toolStandbys can serve as backupsAlways test your backs by recovering

A delayed standby could provide lower RTO in case ofdisasterYou can achieve RPO zero by streaming WALs instead ofusing archive_command

Security first: encrypt your backups

Martín Marqués You forgot to put the WHERE in the DELETE?

Introduction What are backups for Backup types Backups considerations Recovering Finally Question

Conclusions

Take backups!Use some automated toolStandbys can serve as backupsAlways test your backs by recoveringA delayed standby could provide lower RTO in case ofdisaster

You can achieve RPO zero by streaming WALs instead ofusing archive_command

Security first: encrypt your backups

Martín Marqués You forgot to put the WHERE in the DELETE?

Introduction What are backups for Backup types Backups considerations Recovering Finally Question

Conclusions

Take backups!Use some automated toolStandbys can serve as backupsAlways test your backs by recoveringA delayed standby could provide lower RTO in case ofdisasterYou can achieve RPO zero by streaming WALs instead ofusing archive_command

Security first: encrypt your backups

Martín Marqués You forgot to put the WHERE in the DELETE?

Introduction What are backups for Backup types Backups considerations Recovering Finally Question

Conclusions

Take backups!Use some automated toolStandbys can serve as backupsAlways test your backs by recoveringA delayed standby could provide lower RTO in case ofdisasterYou can achieve RPO zero by streaming WALs instead ofusing archive_command

Security first: encrypt your backups

Martín Marqués You forgot to put the WHERE in the DELETE?

Introduction What are backups for Backup types Backups considerations Recovering Finally Question

Content

1 Introduction

2 What are backups for

3 Backup types

4 Backups considerations

5 Recovering

6 Finally

7 Question

Martín Marqués You forgot to put the WHERE in the DELETE?

Introduction What are backups for Backup types Backups considerations Recovering Finally Question

?

Martín Marqués You forgot to put the WHERE in the DELETE?