PostgreSQL Backups the Modern Way Backups the... · 2019-10-10 · PostgreSQL Backups the Modern...

PostgreSQL Backupsthe Modern Way

PGDay Russia 2016St Petersburg, Russia

Magnus Hagander magnus@hagander.net

Magnus HaganderRedpill Linpro

Infrastructure servicesPrincipal database consultant

PostgreSQLCore Team memberCommitterPostgreSQL Europe

So, backups...Do you make them?

BackupsAre not superseded by replicationOr cloudOr containers..

BackupsAre boringBut I'm glad you have them

BackupsWhen did you last restore?

PostgreSQL backupsOk, enough genericWhat about backups in PostgreSQL?

Seen this before?pg_dump options:

­Fc    = custom format­Z     = compression­j     = parallel­a = data only, ­s = schema only­n = schema, ­t = table...    

pg_dumpDon't use for backups

Has other good usecasesToo slow to restoreToo much overheadNo PITRExceptions, of course

Physical backupsBase backupsWith or without log archiveFast restoreFull cluster onlyPlatform specific

Base backups#!/bin/bashset ­e

psql ­U postgres ­q "SELECT pg_start_backup('foo')"

tar cfz /backup/$(date +%Y%m%d) /var/lib/pgsql/data

psql ­U postgres ­q "SELECT pg_stop_backup()"    

Base backupsSo many ways to get that wrong

Spot one?

Base backupsThis used to be the only wayMany scripts around that does itMany of those are broken...

pg_basebackupBase backup over replication protocolSafeError handling and recoveryFor most cases

(we'll cover other options later)

pg_basebackup#!/bin/bashset ­e

pg_basebackup ­D /backup/$(date +%Y%m%d).tar ­Ft ­x    

Needs replicationDefaults need to changeBut for now:

wal_level = hot_standbymax_wal_senders = 5    

local  replication  postgres  peer    

Backup formatsplain

Safe copy of data directoryNot good with multiple tablespaces

tarDestination still a directoryEach tablespace gets one file

base.tar

Transaction logxlog required to restore backupFrom beginning of backup to endIn the log archive, right?

Including xlogAlways use -x or -X to include xlogMakes backup independently consistent

With or without log archiveMay back up xlog twice

Use even with log archive!

Including xlog-X fetch

Fetches xlog at end of backupCan fail if xlog rotated

-X streamReplicates xlog over secondary connectionFewer failure scenariosDoes not work with tar

Backup compressionpg_basebackup -Z

Compression happens in pg_basebackupTar format onlyCPU usageRemote server?

Transfer compressionSSL compression

Much harder these daysssh tunneling

ssh mydbserver ­c "pg_basebackup ­Ft ­D­ ­Z9" > backup.tgz    

That's it!With that, you have backupsThat workAnd are (reasonably) safe

PITRPoint in time recoveryYou all want itA bit more setting up

archive_commandTo use PITR, we use log archivinglike this?

archive_command =  'test ! ­f /mnt/archivedir/%f && cp %p /mnt/archivedir/%f'    

Don't do that!

pg_receivexlogRuns on archive serverUses streaming replicationGenerates log archive

pg_receivexlogMore granular recoverySafe against server restartsCan follow timeline switches on master

pg_receivexlogAlways use with replication slot

As of 9.4But we said modern..

Backups should block

pg_receivexlogpg_receivexlog ­D /log/archive ­h master ­S backup    

Ensure it's restarted!

Backup retentionHow long to keep around?What granularity?...

Backup retentionRecovery needs:

Base backupAll xlog from start to endAll xlog from end to pitr

(that's why we use -x!)

Backup retentionfind is o�en enoughDelete logs older than X, base older than Y

Safe if -x was used!

#!/bin/bash

find /var/backups/basebackup ­type f ­mtime +30 ­print0 |   xargs ­0 ­r /bin/rm

find /var/backups/xlog ­type f ­mtime +7 ­print0 |   xargs ­0 ­r /bin/rm    

Not enough?Handles the simple casesBut has limitationsParticularly in management

Other toolsBarmanpgBackRest

BarmanBackup schedulingLog archivingRetention managementMulti-serverRestore shortcuts

BarmanDeveloped by 2ndQuadrantPythonGPLv3Primarily ssh+rsync

1.6 just learned about pg_receivexlog!Does not use pg_basebackupNo (safe) concurrent backup support

pgBackRestBackup schedulingLog archivingRetention managementMulti-serverRestore shortcuts

pgBackRestDeveloped by CrunchyDataPerlMIT licensessh but not rsync

pgBackRestCustom protocolParallel backup sessionsFull/Differential/Incremental

Segment based

pgBackRestNo pg_receivexlog supportNo concurrent backup supportYet

Summary

Don't roll your own!

Don't roll your ownToo many pitfallsBoth base backups and archivingBackups are too important!

Don't roll your ownPrimary choice

Built-inIf it's enough

Secondary choicepgBackRestBarman

Tertiary choiceRestart from top of slide

Thank you!Magnus Hagander

magnus@hagander.net @magnushagander

http://www.hagander.net/talks/

This material is licensed