Fun Things to do with Logical Decoding

Migrating Rant & Rave to PostgreSQLMike Fowler, [email protected] [email protected]

PGDayUK 2014

Fun Things to do with Logical Decoding

Mike FowlerPGDay UK 2015

Fun Things to do with Logical DecodingMike Fowler, [email protected] [email protected]

PGDayUK 2015

Overview

● What is Logical Decoding?

● Replications Slots

● Output Plugins

● Enabling Logical Decoding

● What does Logical Decoding make possible?


PGDayUK 2015

About Me

● Head of Systems Engineering at Rant & Rave

● Been using PostgreSQL for over 10 years (since 7.4)

● Contributed some XML support

– XMLEXISTS/xpath_exists()

– xml_is_well_formed()

● Contributed a number of bugfixes to the JDBC driver


PGDayUK 2015

What is Logical Decoding?

“The process of extracting all persistent changes to a database's tables into a coherent, easy to understand format which can be interpreted without detailed knowledge of the database's internal state”

“Implemented by decoding the contents of the write-ahead log, which describe changes on a storage level, into an application-specific form such as a stream of tuples or SQL statements”


PGDayUK 2015

No, really - what is Logical Decoding?

● The results of logical decoding give a perspective on the logical changes made during a transaction

● The process of logical decoding allows human understanding of the physical changes made during a transaction

● Achieved by creating a replication slot with a plugin to produce data for a receiver

– Basic example plugin available in contrib

– New utility pg_recvlogical to consume plugin output

– Can roll your own to match your requirements


PGDayUK 2015

Some PostgreSQL Basics

● PostgreSQL is a transactional database

● All changes in PostgreSQL are recorded in a Write Ahead Log (WAL) before any data on disk is actually changed

● The WAL is the key to replication

– Essentially log based binary replication

– Master runs the walsender to transmit new segments

– Slaves run walreceivers to receive the segments


PGDayUK 2015

A Basic Example

We'll use a simple table called sample:

CREATE TABLE sample (

id SERIAL PRIMARY KEY,

data JSON,

captured TIMESTAMP

DEFAULT CURRENT_TIMESTAMP

);


PGDayUK 2015

A Basic Example

INSERT INTO sample (data)

VALUES ('{"beverage":"coffee","temp":96}');

● The backend takes our INSERT and calculates the values for the omitted columns id and captured

● The backend also determines exactly what needs to be changed on disk and commits this to the WAL


PGDayUK 2015

Replication Slots

● 9.4 introduced replication slots

● Capability to retain WAL segments on the server instead of relying on continuous archiving

● Allows master to retain segments even when a slave is offline

● There are two types of replication slot

– Physical

– Logical


PGDayUK 2015

Logical Replication Slots

● Entries in a logical slot are WAL segments that have been decoded by means of an output plugin

● Output plugins are responsible for transforming the data from the WAL into the format the consumer of a logical replication slot desires

● contrib/test_decoding is a working example of the interface and can be used to play


PGDayUK 2015

Output Plugin Rules

● Only transactions that have been successfully flushed to disk are decoded

● Transactions that are rolled back never get decoded

● Changes to UNLOGGED or TEMPORARY tables are also not decoded

● Concurrent transactions are decoded in commit order


PGDayUK 2015

Making an Output Plugin

● Consists of building a shared library that implements a number of functions as detailed in section 46.6 of the manual

– An initialization function

– Transaction begin, row change and transaction end callbacks

– Optional plugin start & stop callbacks

– 9.5 also adds an optional origin filter callback

● For a given transaction the begin & end is called once with the row change called for each changed row even if multiple rows were changed in a single statement


PGDayUK 2015

Output Plugin Considerations

● Keep the plugin simple

– It's running as part of your database

– It's going to process all your changes

● Offload complex filter/analysis to the slot consumer

● Drop the slot if it's no longer in use to stop using server resources

SELECT pg_drop_replication_slot('pgday_slot');


PGDayUK 2015

Enabling Logical Decoding

● Enable replication slots (will require restart)

– wal_level must be set to 'logical'

– max_replication_slots must be at least 1

● Create a replication slotpgday15=# SELECT * FROM pg_create_logical_replication_slot('pgday_slot', 'test_decoding');

slot_name | xlog_position ------------+--------------- pgday_slot | 0/16C9A78(1 row)


PGDayUK 2015

Revisiting the Basic Example

We used a simple table called sample:

CREATE TABLE sample (

id SERIAL PRIMARY KEY,

data JSON,

captured TIMESTAMP

DEFAULT CURRENT_TIMESTAMP

);


PGDayUK 2015

Revisiting the Basic Example

INSERT INTO sample (data)

VALUES ('{"beverage":"coffee","temp":96}');

● The backend took our INSERT and calculated the values for the omitted columns id and captured

● The backend also determined exactly what needed to be changed on disk and committed it to the WAL

● This time the test_decoding plugin was invoked when the WAL segment entered the logical replication slot


PGDayUK 2015

Consuming the Plugin Output

● All data from the plugin will remain in the slot until consumed

● Data can be consumed in two ways

– SQL interface● Useful for experimentation and demonstration

● Can be used to form a basic programmatic integration with non-C applications (e.g. Java using JDBC)

– pg_recvlogical● Similar to pg_receivexlog

● Can be used to output to a file for later consumption


PGDayUK 2015

The Output from our Example

pgday15=# SELECT * FROM pg_logical_slot_get_changes('pgday_slot', NULL, NULL);

location | xid | data

-----------+-----+----------------------------------

0/16C9ED0 | 723 | BEGIN 723

0/16C9ED0 | 723 | table public.sample: INSERT: id[integer]:1 data[json]:'{"beverage":"coffee","temp":96}' captured[timestamp without tz]:'2015-07-03 19:52:39.497161'

0/16CA000 | 723 | COMMIT 723

(3 rows)


PGDayUK 2015

pg_recvlogical

● Use pg_recvlogical to record to a single file

– -f option to specify output file

– -F to specify frequency of fsync in seconds

● Run pg_recvlogical on a separate server

– -h to specify hostname or ip

– -p port

● Can also create/delete slots


PGDayUK 2015

Some simple LD Use Cases

● Trigger-less auditing

– Keep a record of all changes in transaction order

– Change only query log

● Trigger auditing

● Basic throughput analysis


PGDayUK 2015

Limitations of test_decoding

● Only details the new/changed row

● No detail of a WHERE clause

● No timestamps in output to help cross reference

● Changed indexes don't make for easy readingtable public.sample: UPDATE: old-key: id[integer]:3 new-tuple: id[integer]:4 data[json]:'{"beverage":"coffee","temp":96}' captured[timestamp without time zone]:'2015-07-07 00:07:56.305964'


PGDayUK 2015

Enhancing test_decoding

● Plugins can use most of backend infrastructure

– Can use output functions

– Read only access to relations

– Can not use anything that requires a transaction

● Information logged to WAL can be tuned with REPLICA IDENTITY

– Can be adjusted per table with ALTER TABLE

– DEFAULT keeps old value of PRIMARY KEY

– USING INDEX will keep values of columns covered


PGDayUK 2015

Example: decoder_raw

● Enhanced version of test_decoding by Michael Paquier

● Appends WHERE clauses based on data derived from REPLICA IDENTITY

– If DEFAULT, PRIMARY KEY column(s) appear in WHERE

● Removes the commit number from BEGIN and COMMIT

● Available on Git Hub https://github.com/michaelpq/pg_plugins/tree/master/decoder_raw


PGDayUK 2015

decoder_raw Output=# SELECT slot_name FROM pg_create_logical_replication_slot('custom_slot', 'decoder_raw'); slot_name-------------- custom_slot(1 row)=# CREATE TABLE aa (a int primary key, b text);CREATE TABLE=# INSERT INTO aa VALUES (1, 'aa'), (2, 'bb');INSERT 0 2=# UPDATE aa SET b = 'cc' WHERE a = 1;UPDATE 1=# DELETE FROM aa WHERE a = 1;DELETE 1


PGDayUK 2015

decoder_raw Output=# SELECT data FROM pg_logical_slot_peek_changes('custom_slot', NULL, NULL, 'include-transaction', 'on'); data---------------------------------------------------- BEGIN; COMMIT; BEGIN; INSERT INTO public.aa (a, b) VALUES (1, 'aa'); INSERT INTO public.aa (a, b) VALUES (2, 'bb'); COMMIT; BEGIN; UPDATE public.aa SET a = 1, b = 'cc' WHERE a = 1 ; COMMIT; BEGIN; DELETE FROM public.aa WHERE a = 1 ; COMMIT;(12 rows)


PGDayUK 2015

Partial Replication

● Can be achieved in two ways

– Plugin can filter out unneeded tables

– Slot consumer can discard unneeded tables

● Useful if you have write notes independent of a cluster

● Streaming partial replication is possible

– If output of plugin is well structured pg_recvlogical could be piped straight into a psql client


PGDayUK 2015

Full Statement Replication

● Changes can be streamed to other DBMS platforms

– Allows replication to non-PostgreSQL servers

– Useful for keeping a backup during a migration to PostgreSQL

● Differs from most versions of statement replication

– The master computes the statement and determines the changes

– Values of columns have already been computed so will be identical (e.g. CURRENT_TIMESTAMP)


PGDayUK 2015

Conclusion

● Logical Decoding exposes meta-data about committed transactions

● Plugins can be written to enrich and format the meta-data for placement in replication slots

● Data in replication slots can be consumed by external programs using SQL or pg_recvlogical to perform a number of useful tasks


PGDayUK 2014

Questions?



PGDayUK 2014

Thank you!


Software

Fun Things to do with Logical Decoding