Kevin Kempter PostgreSQL Backup and Recovery Methods @ Postgres Open

Consistent State www.consistentstate.com [email protected]

PostgreSQL Backup and Recovery Methods

Kevin Kempter Chief Data Architect

Wednesday, September 18, 13

http://www.consistentstate.com




mailto:[email protected]

mailto:[email protected]

-or-How NOT to be this guy

2


Session Topics✓Overview✓Backup Options✓Restore Options✓Point in Time Recovery

3


Overview• Multiple backup methods• Multiple backup file formats• Many recovery choices / options if a

pg_restore compatible format is used• PITR• PITR is the base construct for WAL

shipping (Warm Standby)4


PostgreSQL Backup Options✓pg_dump✓pg_dumpall

5


pg_dump✓ Utility to dump a snapshot of a single

database

✓ Multiple output options

✓ Non-blocking

✓ Creates a “Consistent” backup - even if the database is in use

6


pg_dump• Syntax

• pg_dump [connection-options] [dump-options] [dbname]

• Connection Options• -h, --host=HOSTNAME • -p, --port=PORT • -U, --username=NAME• -w, --no-password• -W, --password (should happen automatically)• --role=ROLENAME (do SET ROLE before dump)

• Environment Variables• PGDATABASE• PGHOST• PGOPTIONS• PGPORT• PGUSER

7


pg_dump - Common Options• -s [--schema-only]

• Dump schema (DDL) only, no data

• -a [--data-only]• Dump data only - no DDL

• -c [--clean]• Generate drop statements for all created objects

• -C [--create]• Generate a “CREATE DATABASE” statement

• -n schema [--schema=schema]• Only dump the specified schema, wildcard characters are allowed, also multiple -n’s are

allowed

8


pg_dump - Common Options (continued)• -N schema [--exclude-schema=schema]

• Exclude specified schema

• -F format [--format=format]• Output format• p (Plain) plain sql file #default• c (Custom) custom binary format• t (tar) tar format• d (directory) Creates a directory with one file per table/blob, plus a TOC file in a binary

format that pg_restore can read

• -o [--oids]• Dump table OID’s

• -O [--no-owner]• Do not generate ownership commands

9


pg_dump - Common Options (continued)

• -t table [--table=table]• Only dump the specified table, wildcard characters are allowed, also multiple -t’s are

allowed, over-rides -n and -N options

• -x [--no-privileges] [--no-acl]• Do not dump access privileges

• --inserts • Generate INSERT statements

• --disable-triggers• disable triggers during restore (for a data only restore) when doing a data-only dump

• --lock-wait-timeout=timeout• fail if shared lock on an object cannot be acquired within timeout time

10


pg_dump - Common Options (continued)

• -Z 0..9 [--compress=0..9]• Specify compression level for custom format or plain format (not supported for tar

format)

• -v [--verbose]

• -V [--version]

11


pg_dump - Examples

$ pg_dump -C --inserts prod1_db > prod1_db.sqlCreates a dump of insert statements including a create database statement $ pg_dump --data-only --table=customer -Fc prod1_db > prod1_db.cust.fc.dmpDump the customer table data (data only) in a custom format from the prod1_db database

$ pg_dump -s prod1_db > prod1_db.ddl_only.sqlCreates a DDL only dump of the prod1_db database

$ pg_dump --schema=gold -Ft prod1_db > prod1_db.gold_schema.dmpCreates a dump of the gold schema in the prod1_db database in a tar format

12


pg_dump Summary

13


pg_dumpall✓ Utility to dump a snapshot of a full

database cluster (or cluster-wide constructs)

✓ Dumps only to plain sql format

✓ Non-blocking

✓ Creates a “Consistent” backup - even if the database is in use

14


pg_dumpall• Syntax

• pg_dump [connection-options] [dump-options]



15


pg_dumpall - Common Options• -s [--schema-only]

• Dump schema (DDL) only, no data



• -o [--oids]• Dump table OID’s

• -O [--no-owner]• Do not generate ownership commands

16


pg_dumpall - Common Options (continued)• -r [--roles-only]

• Dump only CREATE ROLE data

• -t [--tablespaces-only]• Dump only CREATE TABLESPACE data

• -g [--globals-only]• Dump Global Structures (Roles and Tablespaces)

• --no-tablespaces• Do NOT dump CREATE TABLESPACE Data

• --inserts • Generate INSERT statements

17


pg_dumpall - Common Options (continued)

• --disable-triggers• disable triggers during restore (for a data only restore) when doing a data-only dump

• --lock-wait-timeout=timeout• fail if shared lock on an object cannot be acquired within timeout time

• -v [--verbose]

• -V [--version]

18


pg_dumpall - Examples

$ pg_dumpall -g > prod1_db_cluster.global_structures.sqlCreates a cluster dump containing only the cluster global structures $ pg_dumpall --tablespaces-only > prod1_db_cluster.tablespaces.sqlDump the cluster tablespaces

$ pg_dumpall --no-tablespaces > prod1_db_cluster.no_tablespaces.sqlCreates a dump of the cluster without any tablespace references

$ pg_dumpall -a > prod1_db_cluster.data_only.sqlCreates a dump of the cluster - data only

19


pg_dumpall summary

20


PostgreSQL Restore Options✓psql✓pg_restore

21


Restoring with psql

$ psql -ef prod1_db.sql > load_db.log 2>&1

$ pg_dump prod1_db | psql -h qa_server

$ pg_dumpall -g | psql -h dev_server -p 5433 > load_dev.log 2>&1

$ pg_dump prod1_db | psql -e test_db > load_test_db.log 2>&1

22


pg_restore✓ Utility to restore a data file created by

pg_dump

• Works only with non plain text file formats

23


pg_restore• Syntax

• pg_restore [connection-option...] [option...] [filename]



24


pg_restore - Common Options• -d dbname [--dbname=dbname]

• -C [--create]• Create the specified database before restore


• -s [--schema-only]• Dump schema (DDL) only, no data


25


pg_restore - Common Options (continued)• -n namespace [--schema=schema]

• Restore only objects in the specified schema

• -O [--no-owner]• Do not restore ownership of objects

• -I index [--index=index]• Restore specified index only

• -P function-name(argtype [, ...]) [ --function=function-name(argtype [, ...]) ]• Restore specified function only

• -T trigger [--trigger=trigger]• Restore specified trigger only

26


pg_restore - Common Options (continued)

• -t table [--table=table]• Restore specified table only

• --no-tablespaces• Do not restore any TABLESPACES

• -F format [--format=format]• Output format• c (Custom) custom binary format• t (tar) tar format• d (directory)

27



• -t table [--table=table]• Restore specified table only

• --no-tablespaces• Do not restore any TABLESPACES

• -j number-of-jobs [--jobs=number-of-jobs]• Use parallel jobs to perform the restore

• --disable-triggers• Disable triggers during the restore (for a data only restore)

• -e [--exit-on-error]• Exits upon any error

28



• -l [--list]• Create a list (TOC) file

• -L list-file [--use-list=list-file]• Restore based on the specified list file

• -V [--version]

• -v [--verbose]

29


pg_restore - Examples

$ pg_restore -a -Fc -d prod2_db prod1_db.fc.dmp

Restores data only from a custom formatted file into database prod2_db $ pg_restore -c --schema=gold_partners -v -Ft -d prod2_db prod.tar.dmp

Cleans (removes data & structures first) then restores the gold_partners schema from a tar formatted file into the prod2_db database (with verbose output)

$ pg_restore --schema-only -d qa1_db -Fc -j 10 prod1_db.fc.dmp

Restores the schema only (DDL) from a custom formatted file into the qa1_db database using 10 parallel streams to do the restore

30


Restoring via a list file• pg_restore can create a list file

from a pg_dump file

• List file will contain one line per needed operation such as:

• CREATE TABLE

• COPY

• CREATE INDEX

• List file can be modified as desired to create a custom restore

31


Create a list file from the pg_dump file

$ pg_dump -Ft db1 > db1.fc.dmp

$ pg_restore -Ft -l db1.dmp > db1.lst

32


Sample list file header

;; Archive created at Tue Sep 10 09:42:24 2013; dbname: testdb; TOC Entries: 34; Compression: -1; Dump Version: 1.12-0; Format: CUSTOM; Integer: 4 bytes; Offset: 8 bytes; Dumped from database version: 9.2.4; Dumped by pg_dump version: 9.2.4;;

33


Sample list file contents

; Selected TOC Entries:;1981; 1262 16386 DATABASE - testdb_old postgres6; 2615 2200 SCHEMA - public postgres1982; 0 0 COMMENT - SCHEMA public postgres1983; 0 0 ACL - public postgres181; 3079 11730 EXTENSION - plpgsql1984; 0 0 COMMENT - EXTENSION plpgsql168; 1259 16411 TABLE public testdb_jasper_metrics_tables postgres169; 1259 16414 TABLE public testdb_jasper_metrics_tables_tmp1 postgres170; 1259 16417 TABLE public testdb_metrics_activity postgres171; 1259 16423 TABLE public testdb_metrics_database postgres172; 1259 16426 TABLE public testdb_postgres_metrics_bgwriter postgres173; 1259 16429 TABLE public testdb_postgres_metricsio_user_tables postgres174; 1259 16432 TABLE public testdb_testdb_gf_metrics_tables postgres175; 1259 16435 TABLE public testdb_testdb_transition_metrics_tables postgres176; 1259 16438 TABLE public testdb_testdb_transition_metrics_tables_tmp1 postgres177; 1259 16441 TABLE public testdb_testdb_transition_metrics_tables_tmp2 postgres 34


Sample list file contents (cont)

178; 1259 16444 TABLE public idle_conn_metrics postgres179; 1259 16447 TABLE public total_conn_metrics postgres180; 1259 16450 TABLE public waiting_conn_metrics postgres1964; 0 16411 TABLE DATA public testdb_jasper_metrics_tables postgres1965; 0 16414 TABLE DATA public testdb_jasper_metrics_tables_tmp1 postgres1966; 0 16417 TABLE DATA public testdb_metrics_activity postgres1967; 0 16423 TABLE DATA public testdb_metrics_database postgres1968; 0 16426 TABLE DATA public testdb_postgres_metrics_bgwriter postgres1969; 0 16429 TABLE DATA public testdb_postgres_metricsio_user_tables postgres1970; 0 16432 TABLE DATA public testdb_testdb_gf_metrics_tables postgres1971; 0 16435 TABLE DATA public testdb_testdb_transition_metrics_tables postgres1972; 0 16438 TABLE DATA public testdb_testdb_transition_metrics_tables_tmp1 postgres1973; 0 16441 TABLE DATA public testdb_testdb_transition_metrics_tables_tmp2 postgres1974; 0 16444 TABLE DATA public idle_conn_metrics postgres1975; 0 16447 TABLE DATA public total_conn_metrics postgres1976; 0 16450 TABLE DATA public waiting_conn_metrics postgres

35


Restore via list file - example

$ pg_dump -Ft prod_db > prod_db.fc.dmp

$ pg_restore -Ft -l prod_db.dmp > prod_db.lst

$ createdb qadb3

Edit prod_db.lst as needed / desired

$ pg_restore -L prod_db.lst -Ft -d qadb3 prod_db.dmp

36


Restore Options

Summary

37


Point In Time Recovery(PITR)

38


PITR Overview

•PITR Backups • Archiving the WAL segments• Making Base Backups

•PITR Recovery• Restore the last Base Backup• Prepare the recovered system data directory• Create a recovery.conf file• Start the postmaster

39


PITR Setup• Enable / set the following parameters in the

postgresql.conf file:• wal_level = archive (or hot_standby)• archive_mode = on• archive_command = 'valid archive command'

Can be any valid shell command (including scripts)• archive_timeout = [timeout]

• Special archive_command (and recovery.conf file) tags• %p = full path (absolute path) and the file name of the WAL

segment to be archived• %f = only the file name of the WAL segment• %% = insert a % character in the command string.

40


PITR Example• Enable / set the following parameters in the postgresql.conf file:

• wal_level = archive • archive_mode = on• archive_command = 'cp %p /stage/wal/%f'

Can be any valid shell command (including scripts)• archive_timeout = 0

• mkdir /stage/wal

• chown postgres:postgres /stage/wal

• Re-start the Server

41


PITR Example - create transactions• Execute SQL commands / transactions • Enable access, turn on applications, etc

• This should force the creation of multiple archived WAL files in the /stage/wal directory

• WAL segments are copied when:• The WAL segment is full (see checkpoint_segments)• Number of seconds specified in archive_timeout has passed

42


PITR Example - create base backup

• Execute pg_start_backup$ psql pitr_test # select pg_start_backup ('tag') ;

• Archive the cluster data directory (and any related tablespaces)

$ tar -czvf /backups/pitr/<date>.data.tar.gz ./datarsyncother copy methods

• Execute pg_stop_backup$ psql pitr_test # select pg_stop_backup () ;

43


PITR Example - create more transactions

• Execute SQL commands / transactions

• The application, user connections, etc will continue to generate transactions (and archived WAL segments)

• Verify the creation of additional archived WAL files in the /stage/wal directory

44


PITR - recovery.conf file (common options)

Recovery settings are placed in the file 'recovery.conf'

• restore_command (string) must return nonzero

• restore_command = 'cp /stage/wal/%f %p'

• restore_command = '/usr/local/bin/restore_shell.sh %p %f'

45


PITR - recovery.conf file (common options)

recovery_target_time (timestamp)

• specifies the time stamp up to which recovery will proceed.

• recovery_target_time and recovery_target_xid are mutually exclusive

• The default is to recover to the end of the WAL log.

46


PITR Recovery

(1) If available copy the original cluster data directory to an alternate location

if space is an issue at least copy the old pg_xlog dir it may contain additional unarchived WAL segments

(2) Ensure the postmaster is not running

47


PITR Recovery

If your backup was an rsync to a second server then skip steps 3 & 4

(3) Remove the cluster data directory and any tablespace directories

(4) Restore your last system backup • make sure permissions are retained

• If you're using tablespaces then verify that the symbolic links in pg_tblspc/ were restored

48


PITR Recovery

(5) Remove any wal segments from the pg_xlog dir that were restored from the backup

If you didn't backup pg_xlog then create it, make sure you re-establish it as a symbolic link if needed If needed also re-create the pg_xlog/archive_status directory

(6) Copy the files from the original pg_xlog dir (if available) into the new pg_xlog dir

do a copy as opposed to a move in case you need to start over

49


PITR Recovery

(7) Create a recovery command (recovery.conf) in the cluster data directory.

(8) [Optional] Temporarily modify pg_hba.conf to prevent ordinary users from connecting until the recovery is complete

(9) Start the server. The server will go into recovery mode via the recovery.conf file.

Once the recovery is complete then the server will become available and rename the recovery.conf file to recovery.done

If an error interrupts the recovery (or stops the server) then simply re-starting the server will restart the recovery

50


PITR Recovery

(10) Verify the recovery.If the database was not recovered properly (or to a state that you desire) then go back to step 1

(11) restore the pg_hba.conf to its original state and run a pg_ctl reload (if it was modified for the recovery)

51


Questions?

Thank You!

52


Technology

Kevin Kempter PostgreSQL Backup and Recovery Methods @ Postgres Open