Scaling PostgreSQL with Skytools

Scaling with SkyTools& More

Scaling-Out Postgres with Skype’s Open-Source Toolset

Gavin M. RoySeptember 14th, 2011

About Me

• PostgreSQL ~ 6.5

• CTO @myYearbook.com

• Scaled initial infrastructure

• Not as involved day-to-day database operational and development

• Twitter: @Crad

Scaling?

Concurrency

6am 8am 10am 12pm 2pm 4pm 6pm 8pm 10pm 12am 2am 4am 6am

Hourly breakdown

Req

uest

s pe

r Se

cond

Increasing Size-On-Disk

Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec

Size

in G

B

Scaling andPostgreSQL Behavior

Size on Disk

Tuples, Indexes, Overhead

Table Size+

Size of all combined Indexes

Relations Indexes

Constraints

• Available Memory

• Disk Speed

• IO Bus Speed

Keep it in memory.

Get Fast Disks & I/O.

Process Forking+

Locks

Client Connections

One Connection per Concurrent Request

Apache+PHPOne connection per backend for each pg_connect

PythonOne connection per connection*

ODBCOne connection to Postgres per ODBC connection

Master Process

Stats Collector

Autovacuum

Wall Writer

Wall Writer

Connection Backend Client Connection

Lock Contention?

Each backend for a connected client has to check for locks

Master Process

Stats Collector

Autovacuum

Wall Writer

Wall Writer

Connection Backend Client Connection

Connection Backend Client Connection

New Client Connection?

Access ShareAccess Exclusive

ExclusiveShare

Share Row ExclusiveShare UpdateRow Share

Row Exclusive

Master Process

Stats Collector

Autovacuum

Wall Writer

Wall Writer

Connection Backend Client Connection

Connection Backend Client Connection

Connection Backend Client Connection

...

Too many connections?

Slow performance

250 Apache Backendsx

1 Connection per Backendx

250 Servers=

62,500 Connections

Solvable Problems!

The Trailblazers

Solving Concurrency

pgBouncer

Session Pooling

Transactional Pooling

Statement Pooling

Connection Pooling

Clients Clients Clients

Postgres Server #1

pgBouncer

Postgres Server #2

Postgres Server #3

Tens TensTens

Hundreds HundredsHundreds

Add Local Pooling

Local pgBouncer Local pgBouncer Local pgBouncer

Postgres Server #1

pgBouncer

Postgres Server #2

Postgres Server #3

ClientsClients Clients

Tens TensTens

Hundreds HundredsHundreds

Tens TensTens

Easy to runUsage: pgbouncer [OPTION]... config.ini -d, --daemon Run in background (as a daemon) -R, --restart Do a online restart -q, --quiet Run quietly -v, --verbose Increase verbosity -u, --user=<username> Assume identity of <username> -V, --version Show version -h, --help Show this help screen and exit

userlist.txt

“username” “password”“foo” “bar”

pgbouncer.ini

Specifying Connections[databases]; foodb over unix socketfoodb =

; redirect bardb to bazdb on localhostbardb = host=localhost dbname=bazdb

; access to dest database will go with single userforcedb = host=127.0.0.1 port=300 user=baz password=foo client_encoding=UNICODE datestyle=ISO connect_query='SELECT 1'

Base Daemon Config

[pgbouncer]logfile = pgbouncer.logpidfile = pgbouncer.pid; ip address or * which means all ip-slisten_addr = 127.0.0.1listen_port = 6432; unix socket is also used for -R.;unix_socket_dir = /tmp

Authentication

; any, trust, plain, crypt, md5auth_type = trust#auth_file = 8.0/main/global/pg_authauth_file = etc/userlist.txtadmin_users = user2, someadmin, otheradminstats_users = stats, root

Stats Users?

SHOW HELP|CONFIG|DATABASES|POOLS|CLIENTS|SERVERS|VERSIONSHOW FDS|SOCKETS|ACTIVE_SOCKETS|LISTS|MEM

pgbouncer=# SHOW CLIENTS; type | user | database | state | addr | port | local_addr | local_port | connect_time ------+-------+-----------+--------+-----------+-------+------------+------------+--------------------- C | stats | pgbouncer | active | 127.0.0.1 | 47229 | 127.0.0.1 | 6000 | 2011-09-13 17:55:46

* Truncated columns for display purposes

psql 9.0+ Problem?

psql -U stats -p 6432 pgbouncerpsql: ERROR:  Unknown startup parameter

Add to pgbouncer.ini:

ignore_startup_parameters = application_name

Pooling Behaviorpool_mode = statement

server_check_query = select 1server_check_delay = 10

max_client_conn = 1000default_pool_size = 20

server_connect_timeout = 15server_lifetime = 1200server_idle_timeout = 60

Skytools

Read Only Copy Read Only Copy Read Only Copy Read Only Copy

Load Balancer

pgBouncer

Canonical Database

Clients Clients Clients Clients

Scale-Out Reads

PGQ

The Ticker

ticker.ini [pgqadm] job_name = pgopen_ticker db = dbname=pgopen # how often to run maintenance [seconds] maint_delay = 600 # how often to check for activity [seconds] loop_delay = 0.1 logfile = ~/Source/pgopen_skytools/%(job_name)s.log pidfile = ~/Source/pgopen_skytools/%(job_name)s.pid

Getting PGQ Running

Setup our ticker:

pgqadm.py ticker.ini install

Run the ticker daemon:

pgqadm.py ticker.ini ticker -d

Londiste

replication.ini[londiste]job_name = pgopen_to_destination provider_db = dbname=pgopen subscriber_db = dbname=destination # it will be used as sql ident so no dots/spacespgq_queue_name = pgopen logfile = ~/Source/pgopen_skytools/%(job_name)s.logpidfile = ~/Source/pgopen_skytools/%(job_name)s.pid

Install Londiste

londiste.py replication.ini provider install

londiste.py replication.ini subscriber install

Start Replication Daemon

londiste.py replication.ini replay -d

DDL?

Add the ProviderTables and Sequences

londiste.py replication.ini provider add public.auth_user

Add the SubscriberTables and Sequences

londiste.py replication.ini subscriber add public.auth_user

Great Success!

PL/Proxy

Scale-Out Reads & Writes

A-F Server G-L Server M-R Server S-Z Server

plProxy Server

How does it work?

Simple Remote Connection

CREATE FUNCTION get_user_email(username text)RETURNS SETOF text AS $$ CONNECT 'dbname=remotedb'; SELECT email FROM users WHERE username = $1;$$ LANGUAGE plproxy;

Sharded Request

CREATE FUNCTION get_user_email(username text)RETURNS SETOF text AS $$ CLUSTER “usercluster”; RUN ON hashtext(username);$$ LANGUAGE plproxy;

Sharding Setup

• Need 3 Functions:

• plproxy.get_cluster_partitions(cluster_name text)

• plproxy.get_cluster_version(cluster_name text)

• plproxy.get_cluster_config(in cluster_name text, out key text, out val text)

get_cluster_partitionsCREATE OR REPLACE FUNCTION plproxy.get_cluster_partitions(cluster_name text)RETURNS SETOF text AS $$BEGIN IF cluster_name = 'usercluster' THEN RETURN NEXT 'dbname=part00 host=127.0.0.1'; RETURN NEXT 'dbname=part01 host=127.0.0.1'; RETURN; END IF; RAISE EXCEPTION 'Unknown cluster';END;$$ LANGUAGE plpgsql;

get_cluster_version

CREATE OR REPLACE FUNCTION plproxy.get_cluster_version(cluster_name text)RETURNS int4 AS $$BEGIN IF cluster_name = 'usercluster' THEN RETURN 1; END IF; RAISE EXCEPTION 'Unknown cluster';END;$$ LANGUAGE plpgsql;

get_cluster_configCREATE OR REPLACE FUNCTION plproxy.get_cluster_config( in cluster_name text, out key text, out val text)RETURNS SETOF record AS $$BEGIN -- lets use same config for all clusters key := 'connection_lifetime'; val := 30*60; -- 30m RETURN NEXT; RETURN;END;$$ LANGUAGE plpgsql;

get_cluster_config values

• connection_lifetime

• query_timeout

• disable_binary

• keepalive_idle

• keepalive_interval

• keepalive_count

SQL/MED

SQL/Med Cluster Definition

CREATE SERVER a_cluster FOREIGN DATA WRAPPER plproxy OPTIONS ( connection_lifetime '1800', disable_binary '1', p0 'dbname=part00 hostname=127.0.0.1', p1 'dbname=part01 hostname=127.0.0.1', p2 'dbname=part02 hostname=127.0.0.1', p3 'dbname=part03 hostname=127.0.0.1' );

PLProxy + SQL/Med Behavior

• PL/Proxy will prefer SQL/Med cluster definitions over the plproxy.get_* functions

• PL/Proxy will fallback to plproxy.get_* functions if there are no SQL/Med clusters

SQL/MED User Mapping

CREATE USER MAPPING FOR bob SERVER a_cluster OPTIONS (user 'bob', password 'secret');

CREATE USER MAPPING FOR public SERVER a_cluster OPTIONS (user 'plproxy', password 'foo');

plproxyrc

https://github.com/myYearbook/plproxyrc

• plpgsql based api for table based management of PL/Proxy

• Used to manage complicated PL/Proxy infrastructure @myYearbook

• BSD Licensed

Postgres Server #1

Postgres Server #2

Postgres Server #3

pgBouncer

“Server-to-Server”

Complex PL/Proxy and pgBouncer Environment

Local pgBouncer

Local pgBouncer

Local pgBouncer

Postgres Server #1

pgBouncer

Postgres Server #3

Clients

Clients

Clients pgBouncer

Load Balancer

plProxy Server plProxy Server

Load Balancer

pgBouncer

pgBouncer

Postgres Server #3

Other Tools and Methods?

Questions?