Embed Size (px)
Citation preview
PostgreSQL (System) Administration
SCALE12X - 2014Los Angeles CA
Stephen Frostsfrostsnowmannet
Resonate Inc bull Digital Media bull PostgreSQL bull Hadoop bull techjobsresonateinsightscom bull httpwwwresonateinsightscom
Stephen FrostbullPostgreSQL
bullMajor Contributor CommitterbullImplemented Roles in 83bullColumn-Level Privileges in 84bullContributions to PLpgSQL PostGIS
bullResonate IncbullPrincipal Database EngineerbullOnline Digital Media CompanybullWere Hiring - techjobsresonateinsightscom
Do you readbullplanetpostgresqlorg
AgendabullTermsbullInstallationbullInitial configurationbullGetting connectedbullUsers RolesbullPermissionsbullBackupsbullMonitoringbullExtensions
TermsbullCluster aka Instance
bullOne PG serverbullone postmaster - listens on one portbullOne set of data files (including tablespaces)bullUsersRoles and tablespaces at cluster levelbullReplication at cluster level
Terms (continued)bullTablespace
bullAlternate directoryfilesystem for PG to store databullPerms must be 0700 owned by postgresbullMust explicitly GRANT create rightsbullCan contain objects from any database
bullDatabasebullLives inside a clusterbullSchemas at the database level
bullSchemabullLives inside a database
bullTables views functions at the schema levelbullDefault public schema allows anyone to create
InstallationbullDebianUbuntuetc
bullaptpostgresqlorgbullAdd PGDG sourceslistd
bullRedHatCentOSetcbullyumpostgresqlorgbullDownload amp Install PGDG RPM
bullMultiple Major Versions
Debian InstallbullConfigs in etcpostgresqlXYmainbullInitial DB in varlibpostgresqlXYmainbullBinaries into usrlibpostgresqlXYbinbullLogs into varlogpostgresqlbullStartup logs in varlogpostgresql alsobullOne init script starts all major versions
Debian ClustersbullDebian provides wrappers and helper scriptsbullpg_lsclusters - lists all PG clustersbullpg_ctlcluster - Control specific clustersbull--cluster option - Specify specific cluster
bullpsql --cluster 92mainbullpg_dump --cluster 92main etc
RedHat InstallbullConfigs in data directorybullDefault DB in varlibpgsqlXYdatabullCreate DB with service postgresql-92 initdbbullBinaries into usrpgsql-XYbinbullLogs into varlibpgsql-XYdatapg_logbullStartup logs in varlibpgsql-XYpgstartuplogbullInit script per major version
PostgreSQL Data DirectorybullSome thing in here do not react well to bulletsbullOn Debian just stay out of itbullOn RedHat be careful to only modify
bullpostgresqlconfbullpg_hbaconfbullpg_identconfbullpg_log
bullDo NOT touch files in pg_xlog or other dirsbullpg_xlog is PGs WAL- not just normal log files
Initial postgresqlconfbulllisten_addresses = (for external access)bullcheckpoint_segments = 30+
bullUses more disk space in pg_xlogbullNever let that partition run out of space
bullcheckpoint_completion_target = 09bullTargets finishing in 90 of time given
bulleffective_cache_size = half the RAMbullNever allocated just for planning
bullmax_wal_senders = 3bullMore later
Loggingbullpostgresqlconf
bulllog_connections = onbulllog_disconnections = onbullline_prefix= m [p] q [l-1] d ur a bulllog_lock_waits = onbulllog_statement = ddlbulllog_min_duration_statement = 100bulllog_temp_files = 0bulllog_autovacuum_min_duration = 0
pg_hbaconfbullControls how users are authenticatedlocal DATABASE USER METHOD [OPTIONS]host DATABASE USER ADDRESS METHOD [OPTIONS]hostssl DATABASE USER ADDRESS METHOD [OPTIONS]hostnossl DATABASE USER ADDRESS METHOD [OPTIONS]
bullRead in order top-to-bottom first match is usedbullhostssl requires SSL connection no is not SSLbullSpecial DBs - all sameuser replicationbullSpecial Users - all + prefix for role membershipbullAddress can be IPv4 or IPv6 can include CIDR maskbullSpecial reject method
Authentication MethodsbullThe ones you should use bullpeer
bullSecure unix-socket-based authbullChecks the Unix username of the user
bullgss (Kerberos)bullIntegreates w MITHeimdal Kerberos and ADbullRecommended for Enterprise deployments
bullcert (SSL Certificate)bullClient-side certificate based authenticationbullUse pg_ident to map CNs to PG usernames
Authentication MethodsbullAcceptable but not idealbullmd5
bullStock usernamepasswordbullUse SSL if youre worried about security
bullpambullModules run as postgres userbullCant be used directly w pam_unixbullsaslauthd can make it work (pam_sasl saslauthd)
bullradiusbullUse SSL if youre worried about security
Auth Method Dontsbulltrust - Never use this- no auth donebullpassword - Password sent in cleartextbullsspi
bullWindows-specificbullUses KerberosGSSAPI underneath
bullidentbullInsecure dont trust it- use peer for local
bullldapbullAuths against an LDAP serverbullUse KerberosGSSAPI if you can
pg_identconfbullDefines mappings which are used in pg_hbamap-name auth-user pg-userkerbnames sfrostSNOWMANNET sfrostcertname stephenfrost sfrost
bullExternal-user to PG-user mappingsbullUnix user joe can be PG user bobbullRegexps can be used- but be carefulbullAlso works for Kerberos client certs etc
Debian configsbullExtra config files in DebianUbuntubullstartconf
bullControls start of this clusterbullCan be auto manual disabled
bullpg_ctlconfbullOptions to pass to pg_ctlbullGenerally dont need to modify it
bullenvironmentbullControlls environment PG starts inbullGenerally dont need to modify it
RedHat configsbullBasically just the initd scripts
Connectingbullsudo su - postgresbullpsqlbull to see backslash-commandsbullh to get help on SQL queriescommandsbullExit with q or ctrl-dbullpsql -h localhost
Looking aroundbulltable pg_stat_activity - aka wbulll - list databases Name | Owner | Encoding | Collate | Ctype | Access privileges-----------+----------+----------+-------------+-------------+----------------------- postgres | postgres | UTF8 | en_USUTF-8 | en_USUTF-8 | template0 | postgres | UTF8 | en_USUTF-8 | en_USUTF-8 | =cpostgres + template1 | postgres | UTF8 | en_USUTF-8 | en_USUTF-8 | =cpostgres +
bulldn - list schemas Name | Owner--------+---------- public | postgres
bulldb - list tablespaces Name | Owner | Location------------+----------+---------- pg_default | postgres | pg_global | postgres |
User setupsbullcreateuser CREATE USERbullpassword to set passwordsbullPrivileges
bullSuperuser- Do not give this outbullCreateRole- Creation and modification of rolesbullCreateDatabase- Allows database creationbullLogin- Allows user to connect to DBbullReplication- Only for replicationsystem userbullAdmin- Allows changing role membershipsbullInherit- Automatically get privileges
RolesbullUsers are really rolesbullGroups are implemented with rolesbullCREATE ROLE (or just createuser --nologin)
bullSame privilege optionsbullCan start as nologin then be granted loginbullCan cascade
bullAny role can be GRANTd to any other rolebullInherit is default acts like group privsbullNoinherit means user must run set role ala sudo
Permissionsbullpublic means all usersbullGRANT REVOKE to givetake away privs roles etcbullCONNECT privs on the database (public by default)bullschemas - CREATE USAGE
bullrecommend dropping public or revoke CREATEbullUse per-user or per-app schemas
bulltables - SELECTINSERTUPDATEDELETETRUNCATEbullview - same (incl update) execute as view ownerbullcolumns - SELECTINSERTUPDATEbullfunctions - SECURITY DEFINER are akin to setuid
Default permsbullGenerally secure-by-default
bullExcept functions- EXECUTE granted by defaultbullOwners have all rights on their objectsbullMembership in owning role == ownership
bullALTER DEFAULT PRIVILEGES - for rolesbullFOR ROLE IN SCHEMA GRANTbullCant be applied to just a schema
bullGRANT ON ALL IN SCHEMAbullFor tables views sequences functionsbullOne-time operation new tables will not have privs
TuningbullFor a dedicated serverbullshared_buffers
bullWill be dedicated to PG for cacheingbullUp to half of main memorybullTry 2G on larger servers more may not helpbullPre-93 need to bump sysctl paramsbullPost-93 you dontbullDefaults to 128MB
Tuning (continued)bullwork_mem
bullUsed for in-memory hashing sorts etcbullCan be increased inside a given connectionbullUsed many times over- not a hard limitbullPer connection so be carefulbullDefaults to 1MB (wayy too small)
bullmaintenance_work_membullUsed for building indexesbullMake it larger before building an indexbullDefaults to 16MB (thats a very small index)
Tuning (continued)bulleffective_cache_size
bullTells PG how much of the DB is in memorybullHalf of main memorybullNever allocated only for planning purposesbullDefaults to 128MB
bullautovacuumbullOn a high-rate server make it more aggressivebullIncrease max_workersbullDecrease autovacuum_vacuum_cost_delaybullDefaults are for lightly loaded systems
Config Bump-Upsbullmax_connections = 100
bullConsider using pg_bouncerbull connections == of CPUs is ideal
bullshared_buffers = couple gigbullProbably not more than 3-4G (Test)
bullmaintenance_work_mem = maybe a gigbullUsed for building indexes
bullmax_locks_per_transaction = 128bullMore if you have lots of objectsbull locks available is actually this max_conn
BackupsbullExtremely importantbullpg_basebackup w WAL recieve
bullBinary-based backupbullMUST have WAL files backed up alsobullNeeds to connect to replication DB
bullpg_dumpbullLogical text-based backupbullDoes not back up indexes must rebuildbullRequires lightweight locks on everything
bullTest restoring your data
Monitoringbullcheck_postgresplbullUseful with Nagios Icinga MRTG etcbullProvides metrics as well as monitoringbullAllows custom query for monitoringbullMinimum set of checksarchive_ready (if doing WAL archiving) --- Number of WAL ready filesautovac_freeze --- How close to Autovacuum Max Freezebackends (Metric) --- Number of Backends runningdbstats (Metrics) --- Lots of different statslistener (If using LISTENNOTIFY) --- Checks if anyone is LISTENinglocks (Metric) --- Number of locks heldpgbouncer options (if using pgbouncer) --- Various pgbouncer checkstxn_idle --- Transactions idle for X timetxn_time --- Transactions longer than X timetxn_wraparound --- How close to transaction wraparound
ExtensionsbullInstall -contrib packagebullUse PGXN - httppgxnorgbulltable pg_available_extensions name | default_version | installed_version | comment--------------------+-----------------+-------------------+---------------------------------------------------------------------- file_fdw | 10 | | foreign-data wrapper for flat file access dblink | 10 | | connect to other PostgreSQL databases from within a database plpgsql | 10 | 10 | PLpgSQL procedural language pg_trgm | 10 | | text similarity measurement and index searching based on trigrams adminpack | 10 | | administrative functions for PostgreSQL ip4r | 20 | | hstore | 11 | | data type for storing sets of (key value) pairs
bulladminpack allows superuser to change anythingbulldx lists installed extensions
Thank youStephen Frost
sfrostsnowmannetnet_snow
Stephen FrostbullPostgreSQL
bullMajor Contributor CommitterbullImplemented Roles in 83bullColumn-Level Privileges in 84bullContributions to PLpgSQL PostGIS
bullResonate IncbullPrincipal Database EngineerbullOnline Digital Media CompanybullWere Hiring - techjobsresonateinsightscom
Do you readbullplanetpostgresqlorg
AgendabullTermsbullInstallationbullInitial configurationbullGetting connectedbullUsers RolesbullPermissionsbullBackupsbullMonitoringbullExtensions
TermsbullCluster aka Instance
bullOne PG serverbullone postmaster - listens on one portbullOne set of data files (including tablespaces)bullUsersRoles and tablespaces at cluster levelbullReplication at cluster level
Terms (continued)bullTablespace
bullAlternate directoryfilesystem for PG to store databullPerms must be 0700 owned by postgresbullMust explicitly GRANT create rightsbullCan contain objects from any database
bullDatabasebullLives inside a clusterbullSchemas at the database level
bullSchemabullLives inside a database
bullTables views functions at the schema levelbullDefault public schema allows anyone to create
InstallationbullDebianUbuntuetc
bullaptpostgresqlorgbullAdd PGDG sourceslistd
bullRedHatCentOSetcbullyumpostgresqlorgbullDownload amp Install PGDG RPM
bullMultiple Major Versions
Debian InstallbullConfigs in etcpostgresqlXYmainbullInitial DB in varlibpostgresqlXYmainbullBinaries into usrlibpostgresqlXYbinbullLogs into varlogpostgresqlbullStartup logs in varlogpostgresql alsobullOne init script starts all major versions
Debian ClustersbullDebian provides wrappers and helper scriptsbullpg_lsclusters - lists all PG clustersbullpg_ctlcluster - Control specific clustersbull--cluster option - Specify specific cluster
bullpsql --cluster 92mainbullpg_dump --cluster 92main etc
RedHat InstallbullConfigs in data directorybullDefault DB in varlibpgsqlXYdatabullCreate DB with service postgresql-92 initdbbullBinaries into usrpgsql-XYbinbullLogs into varlibpgsql-XYdatapg_logbullStartup logs in varlibpgsql-XYpgstartuplogbullInit script per major version
PostgreSQL Data DirectorybullSome thing in here do not react well to bulletsbullOn Debian just stay out of itbullOn RedHat be careful to only modify
bullpostgresqlconfbullpg_hbaconfbullpg_identconfbullpg_log
bullDo NOT touch files in pg_xlog or other dirsbullpg_xlog is PGs WAL- not just normal log files
Initial postgresqlconfbulllisten_addresses = (for external access)bullcheckpoint_segments = 30+
bullUses more disk space in pg_xlogbullNever let that partition run out of space
bullcheckpoint_completion_target = 09bullTargets finishing in 90 of time given
bulleffective_cache_size = half the RAMbullNever allocated just for planning
bullmax_wal_senders = 3bullMore later
Loggingbullpostgresqlconf
bulllog_connections = onbulllog_disconnections = onbullline_prefix= m [p] q [l-1] d ur a bulllog_lock_waits = onbulllog_statement = ddlbulllog_min_duration_statement = 100bulllog_temp_files = 0bulllog_autovacuum_min_duration = 0
pg_hbaconfbullControls how users are authenticatedlocal DATABASE USER METHOD [OPTIONS]host DATABASE USER ADDRESS METHOD [OPTIONS]hostssl DATABASE USER ADDRESS METHOD [OPTIONS]hostnossl DATABASE USER ADDRESS METHOD [OPTIONS]
bullRead in order top-to-bottom first match is usedbullhostssl requires SSL connection no is not SSLbullSpecial DBs - all sameuser replicationbullSpecial Users - all + prefix for role membershipbullAddress can be IPv4 or IPv6 can include CIDR maskbullSpecial reject method
Authentication MethodsbullThe ones you should use bullpeer
bullSecure unix-socket-based authbullChecks the Unix username of the user
bullgss (Kerberos)bullIntegreates w MITHeimdal Kerberos and ADbullRecommended for Enterprise deployments
bullcert (SSL Certificate)bullClient-side certificate based authenticationbullUse pg_ident to map CNs to PG usernames
Authentication MethodsbullAcceptable but not idealbullmd5
bullStock usernamepasswordbullUse SSL if youre worried about security
bullpambullModules run as postgres userbullCant be used directly w pam_unixbullsaslauthd can make it work (pam_sasl saslauthd)
bullradiusbullUse SSL if youre worried about security
Auth Method Dontsbulltrust - Never use this- no auth donebullpassword - Password sent in cleartextbullsspi
bullWindows-specificbullUses KerberosGSSAPI underneath
bullidentbullInsecure dont trust it- use peer for local
bullldapbullAuths against an LDAP serverbullUse KerberosGSSAPI if you can
pg_identconfbullDefines mappings which are used in pg_hbamap-name auth-user pg-userkerbnames sfrostSNOWMANNET sfrostcertname stephenfrost sfrost
bullExternal-user to PG-user mappingsbullUnix user joe can be PG user bobbullRegexps can be used- but be carefulbullAlso works for Kerberos client certs etc
Debian configsbullExtra config files in DebianUbuntubullstartconf
bullControls start of this clusterbullCan be auto manual disabled
bullpg_ctlconfbullOptions to pass to pg_ctlbullGenerally dont need to modify it
bullenvironmentbullControlls environment PG starts inbullGenerally dont need to modify it
RedHat configsbullBasically just the initd scripts
Connectingbullsudo su - postgresbullpsqlbull to see backslash-commandsbullh to get help on SQL queriescommandsbullExit with q or ctrl-dbullpsql -h localhost
Looking aroundbulltable pg_stat_activity - aka wbulll - list databases Name | Owner | Encoding | Collate | Ctype | Access privileges-----------+----------+----------+-------------+-------------+----------------------- postgres | postgres | UTF8 | en_USUTF-8 | en_USUTF-8 | template0 | postgres | UTF8 | en_USUTF-8 | en_USUTF-8 | =cpostgres + template1 | postgres | UTF8 | en_USUTF-8 | en_USUTF-8 | =cpostgres +
bulldn - list schemas Name | Owner--------+---------- public | postgres
bulldb - list tablespaces Name | Owner | Location------------+----------+---------- pg_default | postgres | pg_global | postgres |
User setupsbullcreateuser CREATE USERbullpassword to set passwordsbullPrivileges
bullSuperuser- Do not give this outbullCreateRole- Creation and modification of rolesbullCreateDatabase- Allows database creationbullLogin- Allows user to connect to DBbullReplication- Only for replicationsystem userbullAdmin- Allows changing role membershipsbullInherit- Automatically get privileges
RolesbullUsers are really rolesbullGroups are implemented with rolesbullCREATE ROLE (or just createuser --nologin)
bullSame privilege optionsbullCan start as nologin then be granted loginbullCan cascade
bullAny role can be GRANTd to any other rolebullInherit is default acts like group privsbullNoinherit means user must run set role ala sudo
Permissionsbullpublic means all usersbullGRANT REVOKE to givetake away privs roles etcbullCONNECT privs on the database (public by default)bullschemas - CREATE USAGE
bullrecommend dropping public or revoke CREATEbullUse per-user or per-app schemas
bulltables - SELECTINSERTUPDATEDELETETRUNCATEbullview - same (incl update) execute as view ownerbullcolumns - SELECTINSERTUPDATEbullfunctions - SECURITY DEFINER are akin to setuid
Default permsbullGenerally secure-by-default
bullExcept functions- EXECUTE granted by defaultbullOwners have all rights on their objectsbullMembership in owning role == ownership
bullALTER DEFAULT PRIVILEGES - for rolesbullFOR ROLE IN SCHEMA GRANTbullCant be applied to just a schema
bullGRANT ON ALL IN SCHEMAbullFor tables views sequences functionsbullOne-time operation new tables will not have privs
TuningbullFor a dedicated serverbullshared_buffers
bullWill be dedicated to PG for cacheingbullUp to half of main memorybullTry 2G on larger servers more may not helpbullPre-93 need to bump sysctl paramsbullPost-93 you dontbullDefaults to 128MB
Tuning (continued)bullwork_mem
bullUsed for in-memory hashing sorts etcbullCan be increased inside a given connectionbullUsed many times over- not a hard limitbullPer connection so be carefulbullDefaults to 1MB (wayy too small)
bullmaintenance_work_membullUsed for building indexesbullMake it larger before building an indexbullDefaults to 16MB (thats a very small index)
Tuning (continued)bulleffective_cache_size
bullTells PG how much of the DB is in memorybullHalf of main memorybullNever allocated only for planning purposesbullDefaults to 128MB
bullautovacuumbullOn a high-rate server make it more aggressivebullIncrease max_workersbullDecrease autovacuum_vacuum_cost_delaybullDefaults are for lightly loaded systems
Config Bump-Upsbullmax_connections = 100
bullConsider using pg_bouncerbull connections == of CPUs is ideal
bullshared_buffers = couple gigbullProbably not more than 3-4G (Test)
bullmaintenance_work_mem = maybe a gigbullUsed for building indexes
bullmax_locks_per_transaction = 128bullMore if you have lots of objectsbull locks available is actually this max_conn
BackupsbullExtremely importantbullpg_basebackup w WAL recieve
bullBinary-based backupbullMUST have WAL files backed up alsobullNeeds to connect to replication DB
bullpg_dumpbullLogical text-based backupbullDoes not back up indexes must rebuildbullRequires lightweight locks on everything
bullTest restoring your data
Monitoringbullcheck_postgresplbullUseful with Nagios Icinga MRTG etcbullProvides metrics as well as monitoringbullAllows custom query for monitoringbullMinimum set of checksarchive_ready (if doing WAL archiving) --- Number of WAL ready filesautovac_freeze --- How close to Autovacuum Max Freezebackends (Metric) --- Number of Backends runningdbstats (Metrics) --- Lots of different statslistener (If using LISTENNOTIFY) --- Checks if anyone is LISTENinglocks (Metric) --- Number of locks heldpgbouncer options (if using pgbouncer) --- Various pgbouncer checkstxn_idle --- Transactions idle for X timetxn_time --- Transactions longer than X timetxn_wraparound --- How close to transaction wraparound
ExtensionsbullInstall -contrib packagebullUse PGXN - httppgxnorgbulltable pg_available_extensions name | default_version | installed_version | comment--------------------+-----------------+-------------------+---------------------------------------------------------------------- file_fdw | 10 | | foreign-data wrapper for flat file access dblink | 10 | | connect to other PostgreSQL databases from within a database plpgsql | 10 | 10 | PLpgSQL procedural language pg_trgm | 10 | | text similarity measurement and index searching based on trigrams adminpack | 10 | | administrative functions for PostgreSQL ip4r | 20 | | hstore | 11 | | data type for storing sets of (key value) pairs
bulladminpack allows superuser to change anythingbulldx lists installed extensions
Thank youStephen Frost
sfrostsnowmannetnet_snow
Do you readbullplanetpostgresqlorg
AgendabullTermsbullInstallationbullInitial configurationbullGetting connectedbullUsers RolesbullPermissionsbullBackupsbullMonitoringbullExtensions
TermsbullCluster aka Instance
bullOne PG serverbullone postmaster - listens on one portbullOne set of data files (including tablespaces)bullUsersRoles and tablespaces at cluster levelbullReplication at cluster level
Terms (continued)bullTablespace
bullAlternate directoryfilesystem for PG to store databullPerms must be 0700 owned by postgresbullMust explicitly GRANT create rightsbullCan contain objects from any database
bullDatabasebullLives inside a clusterbullSchemas at the database level
bullSchemabullLives inside a database
bullTables views functions at the schema levelbullDefault public schema allows anyone to create
InstallationbullDebianUbuntuetc
bullaptpostgresqlorgbullAdd PGDG sourceslistd
bullRedHatCentOSetcbullyumpostgresqlorgbullDownload amp Install PGDG RPM
bullMultiple Major Versions
Debian InstallbullConfigs in etcpostgresqlXYmainbullInitial DB in varlibpostgresqlXYmainbullBinaries into usrlibpostgresqlXYbinbullLogs into varlogpostgresqlbullStartup logs in varlogpostgresql alsobullOne init script starts all major versions
Debian ClustersbullDebian provides wrappers and helper scriptsbullpg_lsclusters - lists all PG clustersbullpg_ctlcluster - Control specific clustersbull--cluster option - Specify specific cluster
bullpsql --cluster 92mainbullpg_dump --cluster 92main etc
RedHat InstallbullConfigs in data directorybullDefault DB in varlibpgsqlXYdatabullCreate DB with service postgresql-92 initdbbullBinaries into usrpgsql-XYbinbullLogs into varlibpgsql-XYdatapg_logbullStartup logs in varlibpgsql-XYpgstartuplogbullInit script per major version
PostgreSQL Data DirectorybullSome thing in here do not react well to bulletsbullOn Debian just stay out of itbullOn RedHat be careful to only modify
bullpostgresqlconfbullpg_hbaconfbullpg_identconfbullpg_log
bullDo NOT touch files in pg_xlog or other dirsbullpg_xlog is PGs WAL- not just normal log files
Initial postgresqlconfbulllisten_addresses = (for external access)bullcheckpoint_segments = 30+
bullUses more disk space in pg_xlogbullNever let that partition run out of space
bullcheckpoint_completion_target = 09bullTargets finishing in 90 of time given
bulleffective_cache_size = half the RAMbullNever allocated just for planning
bullmax_wal_senders = 3bullMore later
Loggingbullpostgresqlconf
bulllog_connections = onbulllog_disconnections = onbullline_prefix= m [p] q [l-1] d ur a bulllog_lock_waits = onbulllog_statement = ddlbulllog_min_duration_statement = 100bulllog_temp_files = 0bulllog_autovacuum_min_duration = 0
pg_hbaconfbullControls how users are authenticatedlocal DATABASE USER METHOD [OPTIONS]host DATABASE USER ADDRESS METHOD [OPTIONS]hostssl DATABASE USER ADDRESS METHOD [OPTIONS]hostnossl DATABASE USER ADDRESS METHOD [OPTIONS]
bullRead in order top-to-bottom first match is usedbullhostssl requires SSL connection no is not SSLbullSpecial DBs - all sameuser replicationbullSpecial Users - all + prefix for role membershipbullAddress can be IPv4 or IPv6 can include CIDR maskbullSpecial reject method
Authentication MethodsbullThe ones you should use bullpeer
bullSecure unix-socket-based authbullChecks the Unix username of the user
bullgss (Kerberos)bullIntegreates w MITHeimdal Kerberos and ADbullRecommended for Enterprise deployments
bullcert (SSL Certificate)bullClient-side certificate based authenticationbullUse pg_ident to map CNs to PG usernames
Authentication MethodsbullAcceptable but not idealbullmd5
bullStock usernamepasswordbullUse SSL if youre worried about security
bullpambullModules run as postgres userbullCant be used directly w pam_unixbullsaslauthd can make it work (pam_sasl saslauthd)
bullradiusbullUse SSL if youre worried about security
Auth Method Dontsbulltrust - Never use this- no auth donebullpassword - Password sent in cleartextbullsspi
bullWindows-specificbullUses KerberosGSSAPI underneath
bullidentbullInsecure dont trust it- use peer for local
bullldapbullAuths against an LDAP serverbullUse KerberosGSSAPI if you can
pg_identconfbullDefines mappings which are used in pg_hbamap-name auth-user pg-userkerbnames sfrostSNOWMANNET sfrostcertname stephenfrost sfrost
bullExternal-user to PG-user mappingsbullUnix user joe can be PG user bobbullRegexps can be used- but be carefulbullAlso works for Kerberos client certs etc
Debian configsbullExtra config files in DebianUbuntubullstartconf
bullControls start of this clusterbullCan be auto manual disabled
bullpg_ctlconfbullOptions to pass to pg_ctlbullGenerally dont need to modify it
bullenvironmentbullControlls environment PG starts inbullGenerally dont need to modify it
RedHat configsbullBasically just the initd scripts
Connectingbullsudo su - postgresbullpsqlbull to see backslash-commandsbullh to get help on SQL queriescommandsbullExit with q or ctrl-dbullpsql -h localhost
Looking aroundbulltable pg_stat_activity - aka wbulll - list databases Name | Owner | Encoding | Collate | Ctype | Access privileges-----------+----------+----------+-------------+-------------+----------------------- postgres | postgres | UTF8 | en_USUTF-8 | en_USUTF-8 | template0 | postgres | UTF8 | en_USUTF-8 | en_USUTF-8 | =cpostgres + template1 | postgres | UTF8 | en_USUTF-8 | en_USUTF-8 | =cpostgres +
bulldn - list schemas Name | Owner--------+---------- public | postgres
bulldb - list tablespaces Name | Owner | Location------------+----------+---------- pg_default | postgres | pg_global | postgres |
User setupsbullcreateuser CREATE USERbullpassword to set passwordsbullPrivileges
bullSuperuser- Do not give this outbullCreateRole- Creation and modification of rolesbullCreateDatabase- Allows database creationbullLogin- Allows user to connect to DBbullReplication- Only for replicationsystem userbullAdmin- Allows changing role membershipsbullInherit- Automatically get privileges
RolesbullUsers are really rolesbullGroups are implemented with rolesbullCREATE ROLE (or just createuser --nologin)
bullSame privilege optionsbullCan start as nologin then be granted loginbullCan cascade
bullAny role can be GRANTd to any other rolebullInherit is default acts like group privsbullNoinherit means user must run set role ala sudo
Permissionsbullpublic means all usersbullGRANT REVOKE to givetake away privs roles etcbullCONNECT privs on the database (public by default)bullschemas - CREATE USAGE
bullrecommend dropping public or revoke CREATEbullUse per-user or per-app schemas
bulltables - SELECTINSERTUPDATEDELETETRUNCATEbullview - same (incl update) execute as view ownerbullcolumns - SELECTINSERTUPDATEbullfunctions - SECURITY DEFINER are akin to setuid
Default permsbullGenerally secure-by-default
bullExcept functions- EXECUTE granted by defaultbullOwners have all rights on their objectsbullMembership in owning role == ownership
bullALTER DEFAULT PRIVILEGES - for rolesbullFOR ROLE IN SCHEMA GRANTbullCant be applied to just a schema
bullGRANT ON ALL IN SCHEMAbullFor tables views sequences functionsbullOne-time operation new tables will not have privs
TuningbullFor a dedicated serverbullshared_buffers
bullWill be dedicated to PG for cacheingbullUp to half of main memorybullTry 2G on larger servers more may not helpbullPre-93 need to bump sysctl paramsbullPost-93 you dontbullDefaults to 128MB
Tuning (continued)bullwork_mem
bullUsed for in-memory hashing sorts etcbullCan be increased inside a given connectionbullUsed many times over- not a hard limitbullPer connection so be carefulbullDefaults to 1MB (wayy too small)
bullmaintenance_work_membullUsed for building indexesbullMake it larger before building an indexbullDefaults to 16MB (thats a very small index)
Tuning (continued)bulleffective_cache_size
bullTells PG how much of the DB is in memorybullHalf of main memorybullNever allocated only for planning purposesbullDefaults to 128MB
bullautovacuumbullOn a high-rate server make it more aggressivebullIncrease max_workersbullDecrease autovacuum_vacuum_cost_delaybullDefaults are for lightly loaded systems
Config Bump-Upsbullmax_connections = 100
bullConsider using pg_bouncerbull connections == of CPUs is ideal
bullshared_buffers = couple gigbullProbably not more than 3-4G (Test)
bullmaintenance_work_mem = maybe a gigbullUsed for building indexes
bullmax_locks_per_transaction = 128bullMore if you have lots of objectsbull locks available is actually this max_conn
BackupsbullExtremely importantbullpg_basebackup w WAL recieve
bullBinary-based backupbullMUST have WAL files backed up alsobullNeeds to connect to replication DB
bullpg_dumpbullLogical text-based backupbullDoes not back up indexes must rebuildbullRequires lightweight locks on everything
bullTest restoring your data
Monitoringbullcheck_postgresplbullUseful with Nagios Icinga MRTG etcbullProvides metrics as well as monitoringbullAllows custom query for monitoringbullMinimum set of checksarchive_ready (if doing WAL archiving) --- Number of WAL ready filesautovac_freeze --- How close to Autovacuum Max Freezebackends (Metric) --- Number of Backends runningdbstats (Metrics) --- Lots of different statslistener (If using LISTENNOTIFY) --- Checks if anyone is LISTENinglocks (Metric) --- Number of locks heldpgbouncer options (if using pgbouncer) --- Various pgbouncer checkstxn_idle --- Transactions idle for X timetxn_time --- Transactions longer than X timetxn_wraparound --- How close to transaction wraparound
ExtensionsbullInstall -contrib packagebullUse PGXN - httppgxnorgbulltable pg_available_extensions name | default_version | installed_version | comment--------------------+-----------------+-------------------+---------------------------------------------------------------------- file_fdw | 10 | | foreign-data wrapper for flat file access dblink | 10 | | connect to other PostgreSQL databases from within a database plpgsql | 10 | 10 | PLpgSQL procedural language pg_trgm | 10 | | text similarity measurement and index searching based on trigrams adminpack | 10 | | administrative functions for PostgreSQL ip4r | 20 | | hstore | 11 | | data type for storing sets of (key value) pairs
bulladminpack allows superuser to change anythingbulldx lists installed extensions
Thank youStephen Frost
sfrostsnowmannetnet_snow
AgendabullTermsbullInstallationbullInitial configurationbullGetting connectedbullUsers RolesbullPermissionsbullBackupsbullMonitoringbullExtensions
TermsbullCluster aka Instance
bullOne PG serverbullone postmaster - listens on one portbullOne set of data files (including tablespaces)bullUsersRoles and tablespaces at cluster levelbullReplication at cluster level
Terms (continued)bullTablespace
bullAlternate directoryfilesystem for PG to store databullPerms must be 0700 owned by postgresbullMust explicitly GRANT create rightsbullCan contain objects from any database
bullDatabasebullLives inside a clusterbullSchemas at the database level
bullSchemabullLives inside a database
bullTables views functions at the schema levelbullDefault public schema allows anyone to create
InstallationbullDebianUbuntuetc
bullaptpostgresqlorgbullAdd PGDG sourceslistd
bullRedHatCentOSetcbullyumpostgresqlorgbullDownload amp Install PGDG RPM
bullMultiple Major Versions
Debian InstallbullConfigs in etcpostgresqlXYmainbullInitial DB in varlibpostgresqlXYmainbullBinaries into usrlibpostgresqlXYbinbullLogs into varlogpostgresqlbullStartup logs in varlogpostgresql alsobullOne init script starts all major versions
Debian ClustersbullDebian provides wrappers and helper scriptsbullpg_lsclusters - lists all PG clustersbullpg_ctlcluster - Control specific clustersbull--cluster option - Specify specific cluster
bullpsql --cluster 92mainbullpg_dump --cluster 92main etc
RedHat InstallbullConfigs in data directorybullDefault DB in varlibpgsqlXYdatabullCreate DB with service postgresql-92 initdbbullBinaries into usrpgsql-XYbinbullLogs into varlibpgsql-XYdatapg_logbullStartup logs in varlibpgsql-XYpgstartuplogbullInit script per major version
PostgreSQL Data DirectorybullSome thing in here do not react well to bulletsbullOn Debian just stay out of itbullOn RedHat be careful to only modify
bullpostgresqlconfbullpg_hbaconfbullpg_identconfbullpg_log
bullDo NOT touch files in pg_xlog or other dirsbullpg_xlog is PGs WAL- not just normal log files
Initial postgresqlconfbulllisten_addresses = (for external access)bullcheckpoint_segments = 30+
bullUses more disk space in pg_xlogbullNever let that partition run out of space
bullcheckpoint_completion_target = 09bullTargets finishing in 90 of time given
bulleffective_cache_size = half the RAMbullNever allocated just for planning
bullmax_wal_senders = 3bullMore later
Loggingbullpostgresqlconf
bulllog_connections = onbulllog_disconnections = onbullline_prefix= m [p] q [l-1] d ur a bulllog_lock_waits = onbulllog_statement = ddlbulllog_min_duration_statement = 100bulllog_temp_files = 0bulllog_autovacuum_min_duration = 0
pg_hbaconfbullControls how users are authenticatedlocal DATABASE USER METHOD [OPTIONS]host DATABASE USER ADDRESS METHOD [OPTIONS]hostssl DATABASE USER ADDRESS METHOD [OPTIONS]hostnossl DATABASE USER ADDRESS METHOD [OPTIONS]
bullRead in order top-to-bottom first match is usedbullhostssl requires SSL connection no is not SSLbullSpecial DBs - all sameuser replicationbullSpecial Users - all + prefix for role membershipbullAddress can be IPv4 or IPv6 can include CIDR maskbullSpecial reject method
Authentication MethodsbullThe ones you should use bullpeer
bullSecure unix-socket-based authbullChecks the Unix username of the user
bullgss (Kerberos)bullIntegreates w MITHeimdal Kerberos and ADbullRecommended for Enterprise deployments
bullcert (SSL Certificate)bullClient-side certificate based authenticationbullUse pg_ident to map CNs to PG usernames
Authentication MethodsbullAcceptable but not idealbullmd5
bullStock usernamepasswordbullUse SSL if youre worried about security
bullpambullModules run as postgres userbullCant be used directly w pam_unixbullsaslauthd can make it work (pam_sasl saslauthd)
bullradiusbullUse SSL if youre worried about security
Auth Method Dontsbulltrust - Never use this- no auth donebullpassword - Password sent in cleartextbullsspi
bullWindows-specificbullUses KerberosGSSAPI underneath
bullidentbullInsecure dont trust it- use peer for local
bullldapbullAuths against an LDAP serverbullUse KerberosGSSAPI if you can
pg_identconfbullDefines mappings which are used in pg_hbamap-name auth-user pg-userkerbnames sfrostSNOWMANNET sfrostcertname stephenfrost sfrost
bullExternal-user to PG-user mappingsbullUnix user joe can be PG user bobbullRegexps can be used- but be carefulbullAlso works for Kerberos client certs etc
Debian configsbullExtra config files in DebianUbuntubullstartconf
bullControls start of this clusterbullCan be auto manual disabled
bullpg_ctlconfbullOptions to pass to pg_ctlbullGenerally dont need to modify it
bullenvironmentbullControlls environment PG starts inbullGenerally dont need to modify it
RedHat configsbullBasically just the initd scripts
Connectingbullsudo su - postgresbullpsqlbull to see backslash-commandsbullh to get help on SQL queriescommandsbullExit with q or ctrl-dbullpsql -h localhost
Looking aroundbulltable pg_stat_activity - aka wbulll - list databases Name | Owner | Encoding | Collate | Ctype | Access privileges-----------+----------+----------+-------------+-------------+----------------------- postgres | postgres | UTF8 | en_USUTF-8 | en_USUTF-8 | template0 | postgres | UTF8 | en_USUTF-8 | en_USUTF-8 | =cpostgres + template1 | postgres | UTF8 | en_USUTF-8 | en_USUTF-8 | =cpostgres +
bulldn - list schemas Name | Owner--------+---------- public | postgres
bulldb - list tablespaces Name | Owner | Location------------+----------+---------- pg_default | postgres | pg_global | postgres |
User setupsbullcreateuser CREATE USERbullpassword to set passwordsbullPrivileges
bullSuperuser- Do not give this outbullCreateRole- Creation and modification of rolesbullCreateDatabase- Allows database creationbullLogin- Allows user to connect to DBbullReplication- Only for replicationsystem userbullAdmin- Allows changing role membershipsbullInherit- Automatically get privileges
RolesbullUsers are really rolesbullGroups are implemented with rolesbullCREATE ROLE (or just createuser --nologin)
bullSame privilege optionsbullCan start as nologin then be granted loginbullCan cascade
bullAny role can be GRANTd to any other rolebullInherit is default acts like group privsbullNoinherit means user must run set role ala sudo
Permissionsbullpublic means all usersbullGRANT REVOKE to givetake away privs roles etcbullCONNECT privs on the database (public by default)bullschemas - CREATE USAGE
bullrecommend dropping public or revoke CREATEbullUse per-user or per-app schemas
bulltables - SELECTINSERTUPDATEDELETETRUNCATEbullview - same (incl update) execute as view ownerbullcolumns - SELECTINSERTUPDATEbullfunctions - SECURITY DEFINER are akin to setuid
Default permsbullGenerally secure-by-default
bullExcept functions- EXECUTE granted by defaultbullOwners have all rights on their objectsbullMembership in owning role == ownership
bullALTER DEFAULT PRIVILEGES - for rolesbullFOR ROLE IN SCHEMA GRANTbullCant be applied to just a schema
bullGRANT ON ALL IN SCHEMAbullFor tables views sequences functionsbullOne-time operation new tables will not have privs
TuningbullFor a dedicated serverbullshared_buffers
bullWill be dedicated to PG for cacheingbullUp to half of main memorybullTry 2G on larger servers more may not helpbullPre-93 need to bump sysctl paramsbullPost-93 you dontbullDefaults to 128MB
Tuning (continued)bullwork_mem
bullUsed for in-memory hashing sorts etcbullCan be increased inside a given connectionbullUsed many times over- not a hard limitbullPer connection so be carefulbullDefaults to 1MB (wayy too small)
bullmaintenance_work_membullUsed for building indexesbullMake it larger before building an indexbullDefaults to 16MB (thats a very small index)
Tuning (continued)bulleffective_cache_size
bullTells PG how much of the DB is in memorybullHalf of main memorybullNever allocated only for planning purposesbullDefaults to 128MB
bullautovacuumbullOn a high-rate server make it more aggressivebullIncrease max_workersbullDecrease autovacuum_vacuum_cost_delaybullDefaults are for lightly loaded systems
Config Bump-Upsbullmax_connections = 100
bullConsider using pg_bouncerbull connections == of CPUs is ideal
bullshared_buffers = couple gigbullProbably not more than 3-4G (Test)
bullmaintenance_work_mem = maybe a gigbullUsed for building indexes
bullmax_locks_per_transaction = 128bullMore if you have lots of objectsbull locks available is actually this max_conn
BackupsbullExtremely importantbullpg_basebackup w WAL recieve
bullBinary-based backupbullMUST have WAL files backed up alsobullNeeds to connect to replication DB
bullpg_dumpbullLogical text-based backupbullDoes not back up indexes must rebuildbullRequires lightweight locks on everything
bullTest restoring your data
Monitoringbullcheck_postgresplbullUseful with Nagios Icinga MRTG etcbullProvides metrics as well as monitoringbullAllows custom query for monitoringbullMinimum set of checksarchive_ready (if doing WAL archiving) --- Number of WAL ready filesautovac_freeze --- How close to Autovacuum Max Freezebackends (Metric) --- Number of Backends runningdbstats (Metrics) --- Lots of different statslistener (If using LISTENNOTIFY) --- Checks if anyone is LISTENinglocks (Metric) --- Number of locks heldpgbouncer options (if using pgbouncer) --- Various pgbouncer checkstxn_idle --- Transactions idle for X timetxn_time --- Transactions longer than X timetxn_wraparound --- How close to transaction wraparound
ExtensionsbullInstall -contrib packagebullUse PGXN - httppgxnorgbulltable pg_available_extensions name | default_version | installed_version | comment--------------------+-----------------+-------------------+---------------------------------------------------------------------- file_fdw | 10 | | foreign-data wrapper for flat file access dblink | 10 | | connect to other PostgreSQL databases from within a database plpgsql | 10 | 10 | PLpgSQL procedural language pg_trgm | 10 | | text similarity measurement and index searching based on trigrams adminpack | 10 | | administrative functions for PostgreSQL ip4r | 20 | | hstore | 11 | | data type for storing sets of (key value) pairs
bulladminpack allows superuser to change anythingbulldx lists installed extensions
Thank youStephen Frost
sfrostsnowmannetnet_snow
TermsbullCluster aka Instance
bullOne PG serverbullone postmaster - listens on one portbullOne set of data files (including tablespaces)bullUsersRoles and tablespaces at cluster levelbullReplication at cluster level
Terms (continued)bullTablespace
bullAlternate directoryfilesystem for PG to store databullPerms must be 0700 owned by postgresbullMust explicitly GRANT create rightsbullCan contain objects from any database
bullDatabasebullLives inside a clusterbullSchemas at the database level
bullSchemabullLives inside a database
bullTables views functions at the schema levelbullDefault public schema allows anyone to create
InstallationbullDebianUbuntuetc
bullaptpostgresqlorgbullAdd PGDG sourceslistd
bullRedHatCentOSetcbullyumpostgresqlorgbullDownload amp Install PGDG RPM
bullMultiple Major Versions
Debian InstallbullConfigs in etcpostgresqlXYmainbullInitial DB in varlibpostgresqlXYmainbullBinaries into usrlibpostgresqlXYbinbullLogs into varlogpostgresqlbullStartup logs in varlogpostgresql alsobullOne init script starts all major versions
Debian ClustersbullDebian provides wrappers and helper scriptsbullpg_lsclusters - lists all PG clustersbullpg_ctlcluster - Control specific clustersbull--cluster option - Specify specific cluster
bullpsql --cluster 92mainbullpg_dump --cluster 92main etc
RedHat InstallbullConfigs in data directorybullDefault DB in varlibpgsqlXYdatabullCreate DB with service postgresql-92 initdbbullBinaries into usrpgsql-XYbinbullLogs into varlibpgsql-XYdatapg_logbullStartup logs in varlibpgsql-XYpgstartuplogbullInit script per major version
PostgreSQL Data DirectorybullSome thing in here do not react well to bulletsbullOn Debian just stay out of itbullOn RedHat be careful to only modify
bullpostgresqlconfbullpg_hbaconfbullpg_identconfbullpg_log
bullDo NOT touch files in pg_xlog or other dirsbullpg_xlog is PGs WAL- not just normal log files
Initial postgresqlconfbulllisten_addresses = (for external access)bullcheckpoint_segments = 30+
bullUses more disk space in pg_xlogbullNever let that partition run out of space
bullcheckpoint_completion_target = 09bullTargets finishing in 90 of time given
bulleffective_cache_size = half the RAMbullNever allocated just for planning
bullmax_wal_senders = 3bullMore later
Loggingbullpostgresqlconf
bulllog_connections = onbulllog_disconnections = onbullline_prefix= m [p] q [l-1] d ur a bulllog_lock_waits = onbulllog_statement = ddlbulllog_min_duration_statement = 100bulllog_temp_files = 0bulllog_autovacuum_min_duration = 0
pg_hbaconfbullControls how users are authenticatedlocal DATABASE USER METHOD [OPTIONS]host DATABASE USER ADDRESS METHOD [OPTIONS]hostssl DATABASE USER ADDRESS METHOD [OPTIONS]hostnossl DATABASE USER ADDRESS METHOD [OPTIONS]
bullRead in order top-to-bottom first match is usedbullhostssl requires SSL connection no is not SSLbullSpecial DBs - all sameuser replicationbullSpecial Users - all + prefix for role membershipbullAddress can be IPv4 or IPv6 can include CIDR maskbullSpecial reject method
Authentication MethodsbullThe ones you should use bullpeer
bullSecure unix-socket-based authbullChecks the Unix username of the user
bullgss (Kerberos)bullIntegreates w MITHeimdal Kerberos and ADbullRecommended for Enterprise deployments
bullcert (SSL Certificate)bullClient-side certificate based authenticationbullUse pg_ident to map CNs to PG usernames
Authentication MethodsbullAcceptable but not idealbullmd5
bullStock usernamepasswordbullUse SSL if youre worried about security
bullpambullModules run as postgres userbullCant be used directly w pam_unixbullsaslauthd can make it work (pam_sasl saslauthd)
bullradiusbullUse SSL if youre worried about security
Auth Method Dontsbulltrust - Never use this- no auth donebullpassword - Password sent in cleartextbullsspi
bullWindows-specificbullUses KerberosGSSAPI underneath
bullidentbullInsecure dont trust it- use peer for local
bullldapbullAuths against an LDAP serverbullUse KerberosGSSAPI if you can
pg_identconfbullDefines mappings which are used in pg_hbamap-name auth-user pg-userkerbnames sfrostSNOWMANNET sfrostcertname stephenfrost sfrost
bullExternal-user to PG-user mappingsbullUnix user joe can be PG user bobbullRegexps can be used- but be carefulbullAlso works for Kerberos client certs etc
Debian configsbullExtra config files in DebianUbuntubullstartconf
bullControls start of this clusterbullCan be auto manual disabled
bullpg_ctlconfbullOptions to pass to pg_ctlbullGenerally dont need to modify it
bullenvironmentbullControlls environment PG starts inbullGenerally dont need to modify it
RedHat configsbullBasically just the initd scripts
Connectingbullsudo su - postgresbullpsqlbull to see backslash-commandsbullh to get help on SQL queriescommandsbullExit with q or ctrl-dbullpsql -h localhost
Looking aroundbulltable pg_stat_activity - aka wbulll - list databases Name | Owner | Encoding | Collate | Ctype | Access privileges-----------+----------+----------+-------------+-------------+----------------------- postgres | postgres | UTF8 | en_USUTF-8 | en_USUTF-8 | template0 | postgres | UTF8 | en_USUTF-8 | en_USUTF-8 | =cpostgres + template1 | postgres | UTF8 | en_USUTF-8 | en_USUTF-8 | =cpostgres +
bulldn - list schemas Name | Owner--------+---------- public | postgres
bulldb - list tablespaces Name | Owner | Location------------+----------+---------- pg_default | postgres | pg_global | postgres |
User setupsbullcreateuser CREATE USERbullpassword to set passwordsbullPrivileges
bullSuperuser- Do not give this outbullCreateRole- Creation and modification of rolesbullCreateDatabase- Allows database creationbullLogin- Allows user to connect to DBbullReplication- Only for replicationsystem userbullAdmin- Allows changing role membershipsbullInherit- Automatically get privileges
RolesbullUsers are really rolesbullGroups are implemented with rolesbullCREATE ROLE (or just createuser --nologin)
bullSame privilege optionsbullCan start as nologin then be granted loginbullCan cascade
bullAny role can be GRANTd to any other rolebullInherit is default acts like group privsbullNoinherit means user must run set role ala sudo
Permissionsbullpublic means all usersbullGRANT REVOKE to givetake away privs roles etcbullCONNECT privs on the database (public by default)bullschemas - CREATE USAGE
bullrecommend dropping public or revoke CREATEbullUse per-user or per-app schemas
bulltables - SELECTINSERTUPDATEDELETETRUNCATEbullview - same (incl update) execute as view ownerbullcolumns - SELECTINSERTUPDATEbullfunctions - SECURITY DEFINER are akin to setuid
Default permsbullGenerally secure-by-default
bullExcept functions- EXECUTE granted by defaultbullOwners have all rights on their objectsbullMembership in owning role == ownership
bullALTER DEFAULT PRIVILEGES - for rolesbullFOR ROLE IN SCHEMA GRANTbullCant be applied to just a schema
bullGRANT ON ALL IN SCHEMAbullFor tables views sequences functionsbullOne-time operation new tables will not have privs
TuningbullFor a dedicated serverbullshared_buffers
bullWill be dedicated to PG for cacheingbullUp to half of main memorybullTry 2G on larger servers more may not helpbullPre-93 need to bump sysctl paramsbullPost-93 you dontbullDefaults to 128MB
Tuning (continued)bullwork_mem
bullUsed for in-memory hashing sorts etcbullCan be increased inside a given connectionbullUsed many times over- not a hard limitbullPer connection so be carefulbullDefaults to 1MB (wayy too small)
bullmaintenance_work_membullUsed for building indexesbullMake it larger before building an indexbullDefaults to 16MB (thats a very small index)
Tuning (continued)bulleffective_cache_size
bullTells PG how much of the DB is in memorybullHalf of main memorybullNever allocated only for planning purposesbullDefaults to 128MB
bullautovacuumbullOn a high-rate server make it more aggressivebullIncrease max_workersbullDecrease autovacuum_vacuum_cost_delaybullDefaults are for lightly loaded systems
Config Bump-Upsbullmax_connections = 100
bullConsider using pg_bouncerbull connections == of CPUs is ideal
bullshared_buffers = couple gigbullProbably not more than 3-4G (Test)
bullmaintenance_work_mem = maybe a gigbullUsed for building indexes
bullmax_locks_per_transaction = 128bullMore if you have lots of objectsbull locks available is actually this max_conn
BackupsbullExtremely importantbullpg_basebackup w WAL recieve
bullBinary-based backupbullMUST have WAL files backed up alsobullNeeds to connect to replication DB
bullpg_dumpbullLogical text-based backupbullDoes not back up indexes must rebuildbullRequires lightweight locks on everything
bullTest restoring your data
Monitoringbullcheck_postgresplbullUseful with Nagios Icinga MRTG etcbullProvides metrics as well as monitoringbullAllows custom query for monitoringbullMinimum set of checksarchive_ready (if doing WAL archiving) --- Number of WAL ready filesautovac_freeze --- How close to Autovacuum Max Freezebackends (Metric) --- Number of Backends runningdbstats (Metrics) --- Lots of different statslistener (If using LISTENNOTIFY) --- Checks if anyone is LISTENinglocks (Metric) --- Number of locks heldpgbouncer options (if using pgbouncer) --- Various pgbouncer checkstxn_idle --- Transactions idle for X timetxn_time --- Transactions longer than X timetxn_wraparound --- How close to transaction wraparound
ExtensionsbullInstall -contrib packagebullUse PGXN - httppgxnorgbulltable pg_available_extensions name | default_version | installed_version | comment--------------------+-----------------+-------------------+---------------------------------------------------------------------- file_fdw | 10 | | foreign-data wrapper for flat file access dblink | 10 | | connect to other PostgreSQL databases from within a database plpgsql | 10 | 10 | PLpgSQL procedural language pg_trgm | 10 | | text similarity measurement and index searching based on trigrams adminpack | 10 | | administrative functions for PostgreSQL ip4r | 20 | | hstore | 11 | | data type for storing sets of (key value) pairs
bulladminpack allows superuser to change anythingbulldx lists installed extensions
Thank youStephen Frost
sfrostsnowmannetnet_snow
Terms (continued)bullTablespace
bullAlternate directoryfilesystem for PG to store databullPerms must be 0700 owned by postgresbullMust explicitly GRANT create rightsbullCan contain objects from any database
bullDatabasebullLives inside a clusterbullSchemas at the database level
bullSchemabullLives inside a database
bullTables views functions at the schema levelbullDefault public schema allows anyone to create
InstallationbullDebianUbuntuetc
bullaptpostgresqlorgbullAdd PGDG sourceslistd
bullRedHatCentOSetcbullyumpostgresqlorgbullDownload amp Install PGDG RPM
bullMultiple Major Versions
Debian InstallbullConfigs in etcpostgresqlXYmainbullInitial DB in varlibpostgresqlXYmainbullBinaries into usrlibpostgresqlXYbinbullLogs into varlogpostgresqlbullStartup logs in varlogpostgresql alsobullOne init script starts all major versions
Debian ClustersbullDebian provides wrappers and helper scriptsbullpg_lsclusters - lists all PG clustersbullpg_ctlcluster - Control specific clustersbull--cluster option - Specify specific cluster
bullpsql --cluster 92mainbullpg_dump --cluster 92main etc
RedHat InstallbullConfigs in data directorybullDefault DB in varlibpgsqlXYdatabullCreate DB with service postgresql-92 initdbbullBinaries into usrpgsql-XYbinbullLogs into varlibpgsql-XYdatapg_logbullStartup logs in varlibpgsql-XYpgstartuplogbullInit script per major version
PostgreSQL Data DirectorybullSome thing in here do not react well to bulletsbullOn Debian just stay out of itbullOn RedHat be careful to only modify
bullpostgresqlconfbullpg_hbaconfbullpg_identconfbullpg_log
bullDo NOT touch files in pg_xlog or other dirsbullpg_xlog is PGs WAL- not just normal log files
Initial postgresqlconfbulllisten_addresses = (for external access)bullcheckpoint_segments = 30+
bullUses more disk space in pg_xlogbullNever let that partition run out of space
bullcheckpoint_completion_target = 09bullTargets finishing in 90 of time given
bulleffective_cache_size = half the RAMbullNever allocated just for planning
bullmax_wal_senders = 3bullMore later
Loggingbullpostgresqlconf
bulllog_connections = onbulllog_disconnections = onbullline_prefix= m [p] q [l-1] d ur a bulllog_lock_waits = onbulllog_statement = ddlbulllog_min_duration_statement = 100bulllog_temp_files = 0bulllog_autovacuum_min_duration = 0
pg_hbaconfbullControls how users are authenticatedlocal DATABASE USER METHOD [OPTIONS]host DATABASE USER ADDRESS METHOD [OPTIONS]hostssl DATABASE USER ADDRESS METHOD [OPTIONS]hostnossl DATABASE USER ADDRESS METHOD [OPTIONS]
bullRead in order top-to-bottom first match is usedbullhostssl requires SSL connection no is not SSLbullSpecial DBs - all sameuser replicationbullSpecial Users - all + prefix for role membershipbullAddress can be IPv4 or IPv6 can include CIDR maskbullSpecial reject method
Authentication MethodsbullThe ones you should use bullpeer
bullSecure unix-socket-based authbullChecks the Unix username of the user
bullgss (Kerberos)bullIntegreates w MITHeimdal Kerberos and ADbullRecommended for Enterprise deployments
bullcert (SSL Certificate)bullClient-side certificate based authenticationbullUse pg_ident to map CNs to PG usernames
Authentication MethodsbullAcceptable but not idealbullmd5
bullStock usernamepasswordbullUse SSL if youre worried about security
bullpambullModules run as postgres userbullCant be used directly w pam_unixbullsaslauthd can make it work (pam_sasl saslauthd)
bullradiusbullUse SSL if youre worried about security
Auth Method Dontsbulltrust - Never use this- no auth donebullpassword - Password sent in cleartextbullsspi
bullWindows-specificbullUses KerberosGSSAPI underneath
bullidentbullInsecure dont trust it- use peer for local
bullldapbullAuths against an LDAP serverbullUse KerberosGSSAPI if you can
pg_identconfbullDefines mappings which are used in pg_hbamap-name auth-user pg-userkerbnames sfrostSNOWMANNET sfrostcertname stephenfrost sfrost
bullExternal-user to PG-user mappingsbullUnix user joe can be PG user bobbullRegexps can be used- but be carefulbullAlso works for Kerberos client certs etc
Debian configsbullExtra config files in DebianUbuntubullstartconf
bullControls start of this clusterbullCan be auto manual disabled
bullpg_ctlconfbullOptions to pass to pg_ctlbullGenerally dont need to modify it
bullenvironmentbullControlls environment PG starts inbullGenerally dont need to modify it
RedHat configsbullBasically just the initd scripts
Connectingbullsudo su - postgresbullpsqlbull to see backslash-commandsbullh to get help on SQL queriescommandsbullExit with q or ctrl-dbullpsql -h localhost
Looking aroundbulltable pg_stat_activity - aka wbulll - list databases Name | Owner | Encoding | Collate | Ctype | Access privileges-----------+----------+----------+-------------+-------------+----------------------- postgres | postgres | UTF8 | en_USUTF-8 | en_USUTF-8 | template0 | postgres | UTF8 | en_USUTF-8 | en_USUTF-8 | =cpostgres + template1 | postgres | UTF8 | en_USUTF-8 | en_USUTF-8 | =cpostgres +
bulldn - list schemas Name | Owner--------+---------- public | postgres
bulldb - list tablespaces Name | Owner | Location------------+----------+---------- pg_default | postgres | pg_global | postgres |
User setupsbullcreateuser CREATE USERbullpassword to set passwordsbullPrivileges
bullSuperuser- Do not give this outbullCreateRole- Creation and modification of rolesbullCreateDatabase- Allows database creationbullLogin- Allows user to connect to DBbullReplication- Only for replicationsystem userbullAdmin- Allows changing role membershipsbullInherit- Automatically get privileges
RolesbullUsers are really rolesbullGroups are implemented with rolesbullCREATE ROLE (or just createuser --nologin)
bullSame privilege optionsbullCan start as nologin then be granted loginbullCan cascade
bullAny role can be GRANTd to any other rolebullInherit is default acts like group privsbullNoinherit means user must run set role ala sudo
Permissionsbullpublic means all usersbullGRANT REVOKE to givetake away privs roles etcbullCONNECT privs on the database (public by default)bullschemas - CREATE USAGE
bullrecommend dropping public or revoke CREATEbullUse per-user or per-app schemas
bulltables - SELECTINSERTUPDATEDELETETRUNCATEbullview - same (incl update) execute as view ownerbullcolumns - SELECTINSERTUPDATEbullfunctions - SECURITY DEFINER are akin to setuid
Default permsbullGenerally secure-by-default
bullExcept functions- EXECUTE granted by defaultbullOwners have all rights on their objectsbullMembership in owning role == ownership
bullALTER DEFAULT PRIVILEGES - for rolesbullFOR ROLE IN SCHEMA GRANTbullCant be applied to just a schema
bullGRANT ON ALL IN SCHEMAbullFor tables views sequences functionsbullOne-time operation new tables will not have privs
TuningbullFor a dedicated serverbullshared_buffers
bullWill be dedicated to PG for cacheingbullUp to half of main memorybullTry 2G on larger servers more may not helpbullPre-93 need to bump sysctl paramsbullPost-93 you dontbullDefaults to 128MB
Tuning (continued)bullwork_mem
bullUsed for in-memory hashing sorts etcbullCan be increased inside a given connectionbullUsed many times over- not a hard limitbullPer connection so be carefulbullDefaults to 1MB (wayy too small)
bullmaintenance_work_membullUsed for building indexesbullMake it larger before building an indexbullDefaults to 16MB (thats a very small index)
Tuning (continued)bulleffective_cache_size
bullTells PG how much of the DB is in memorybullHalf of main memorybullNever allocated only for planning purposesbullDefaults to 128MB
bullautovacuumbullOn a high-rate server make it more aggressivebullIncrease max_workersbullDecrease autovacuum_vacuum_cost_delaybullDefaults are for lightly loaded systems
Config Bump-Upsbullmax_connections = 100
bullConsider using pg_bouncerbull connections == of CPUs is ideal
bullshared_buffers = couple gigbullProbably not more than 3-4G (Test)
bullmaintenance_work_mem = maybe a gigbullUsed for building indexes
bullmax_locks_per_transaction = 128bullMore if you have lots of objectsbull locks available is actually this max_conn
BackupsbullExtremely importantbullpg_basebackup w WAL recieve
bullBinary-based backupbullMUST have WAL files backed up alsobullNeeds to connect to replication DB
bullpg_dumpbullLogical text-based backupbullDoes not back up indexes must rebuildbullRequires lightweight locks on everything
bullTest restoring your data
Monitoringbullcheck_postgresplbullUseful with Nagios Icinga MRTG etcbullProvides metrics as well as monitoringbullAllows custom query for monitoringbullMinimum set of checksarchive_ready (if doing WAL archiving) --- Number of WAL ready filesautovac_freeze --- How close to Autovacuum Max Freezebackends (Metric) --- Number of Backends runningdbstats (Metrics) --- Lots of different statslistener (If using LISTENNOTIFY) --- Checks if anyone is LISTENinglocks (Metric) --- Number of locks heldpgbouncer options (if using pgbouncer) --- Various pgbouncer checkstxn_idle --- Transactions idle for X timetxn_time --- Transactions longer than X timetxn_wraparound --- How close to transaction wraparound
ExtensionsbullInstall -contrib packagebullUse PGXN - httppgxnorgbulltable pg_available_extensions name | default_version | installed_version | comment--------------------+-----------------+-------------------+---------------------------------------------------------------------- file_fdw | 10 | | foreign-data wrapper for flat file access dblink | 10 | | connect to other PostgreSQL databases from within a database plpgsql | 10 | 10 | PLpgSQL procedural language pg_trgm | 10 | | text similarity measurement and index searching based on trigrams adminpack | 10 | | administrative functions for PostgreSQL ip4r | 20 | | hstore | 11 | | data type for storing sets of (key value) pairs
bulladminpack allows superuser to change anythingbulldx lists installed extensions
Thank youStephen Frost
sfrostsnowmannetnet_snow
bullTables views functions at the schema levelbullDefault public schema allows anyone to create
InstallationbullDebianUbuntuetc
bullaptpostgresqlorgbullAdd PGDG sourceslistd
bullRedHatCentOSetcbullyumpostgresqlorgbullDownload amp Install PGDG RPM
bullMultiple Major Versions
Debian InstallbullConfigs in etcpostgresqlXYmainbullInitial DB in varlibpostgresqlXYmainbullBinaries into usrlibpostgresqlXYbinbullLogs into varlogpostgresqlbullStartup logs in varlogpostgresql alsobullOne init script starts all major versions
Debian ClustersbullDebian provides wrappers and helper scriptsbullpg_lsclusters - lists all PG clustersbullpg_ctlcluster - Control specific clustersbull--cluster option - Specify specific cluster
bullpsql --cluster 92mainbullpg_dump --cluster 92main etc
RedHat InstallbullConfigs in data directorybullDefault DB in varlibpgsqlXYdatabullCreate DB with service postgresql-92 initdbbullBinaries into usrpgsql-XYbinbullLogs into varlibpgsql-XYdatapg_logbullStartup logs in varlibpgsql-XYpgstartuplogbullInit script per major version
PostgreSQL Data DirectorybullSome thing in here do not react well to bulletsbullOn Debian just stay out of itbullOn RedHat be careful to only modify
bullpostgresqlconfbullpg_hbaconfbullpg_identconfbullpg_log
bullDo NOT touch files in pg_xlog or other dirsbullpg_xlog is PGs WAL- not just normal log files
Initial postgresqlconfbulllisten_addresses = (for external access)bullcheckpoint_segments = 30+
bullUses more disk space in pg_xlogbullNever let that partition run out of space
bullcheckpoint_completion_target = 09bullTargets finishing in 90 of time given
bulleffective_cache_size = half the RAMbullNever allocated just for planning
bullmax_wal_senders = 3bullMore later
Loggingbullpostgresqlconf
bulllog_connections = onbulllog_disconnections = onbullline_prefix= m [p] q [l-1] d ur a bulllog_lock_waits = onbulllog_statement = ddlbulllog_min_duration_statement = 100bulllog_temp_files = 0bulllog_autovacuum_min_duration = 0
pg_hbaconfbullControls how users are authenticatedlocal DATABASE USER METHOD [OPTIONS]host DATABASE USER ADDRESS METHOD [OPTIONS]hostssl DATABASE USER ADDRESS METHOD [OPTIONS]hostnossl DATABASE USER ADDRESS METHOD [OPTIONS]
bullRead in order top-to-bottom first match is usedbullhostssl requires SSL connection no is not SSLbullSpecial DBs - all sameuser replicationbullSpecial Users - all + prefix for role membershipbullAddress can be IPv4 or IPv6 can include CIDR maskbullSpecial reject method
Authentication MethodsbullThe ones you should use bullpeer
bullSecure unix-socket-based authbullChecks the Unix username of the user
bullgss (Kerberos)bullIntegreates w MITHeimdal Kerberos and ADbullRecommended for Enterprise deployments
bullcert (SSL Certificate)bullClient-side certificate based authenticationbullUse pg_ident to map CNs to PG usernames
Authentication MethodsbullAcceptable but not idealbullmd5
bullStock usernamepasswordbullUse SSL if youre worried about security
bullpambullModules run as postgres userbullCant be used directly w pam_unixbullsaslauthd can make it work (pam_sasl saslauthd)
bullradiusbullUse SSL if youre worried about security
Auth Method Dontsbulltrust - Never use this- no auth donebullpassword - Password sent in cleartextbullsspi
bullWindows-specificbullUses KerberosGSSAPI underneath
bullidentbullInsecure dont trust it- use peer for local
bullldapbullAuths against an LDAP serverbullUse KerberosGSSAPI if you can
pg_identconfbullDefines mappings which are used in pg_hbamap-name auth-user pg-userkerbnames sfrostSNOWMANNET sfrostcertname stephenfrost sfrost
bullExternal-user to PG-user mappingsbullUnix user joe can be PG user bobbullRegexps can be used- but be carefulbullAlso works for Kerberos client certs etc
Debian configsbullExtra config files in DebianUbuntubullstartconf
bullControls start of this clusterbullCan be auto manual disabled
bullpg_ctlconfbullOptions to pass to pg_ctlbullGenerally dont need to modify it
bullenvironmentbullControlls environment PG starts inbullGenerally dont need to modify it
RedHat configsbullBasically just the initd scripts
Connectingbullsudo su - postgresbullpsqlbull to see backslash-commandsbullh to get help on SQL queriescommandsbullExit with q or ctrl-dbullpsql -h localhost
Looking aroundbulltable pg_stat_activity - aka wbulll - list databases Name | Owner | Encoding | Collate | Ctype | Access privileges-----------+----------+----------+-------------+-------------+----------------------- postgres | postgres | UTF8 | en_USUTF-8 | en_USUTF-8 | template0 | postgres | UTF8 | en_USUTF-8 | en_USUTF-8 | =cpostgres + template1 | postgres | UTF8 | en_USUTF-8 | en_USUTF-8 | =cpostgres +
bulldn - list schemas Name | Owner--------+---------- public | postgres
bulldb - list tablespaces Name | Owner | Location------------+----------+---------- pg_default | postgres | pg_global | postgres |
User setupsbullcreateuser CREATE USERbullpassword to set passwordsbullPrivileges
bullSuperuser- Do not give this outbullCreateRole- Creation and modification of rolesbullCreateDatabase- Allows database creationbullLogin- Allows user to connect to DBbullReplication- Only for replicationsystem userbullAdmin- Allows changing role membershipsbullInherit- Automatically get privileges
RolesbullUsers are really rolesbullGroups are implemented with rolesbullCREATE ROLE (or just createuser --nologin)
bullSame privilege optionsbullCan start as nologin then be granted loginbullCan cascade
bullAny role can be GRANTd to any other rolebullInherit is default acts like group privsbullNoinherit means user must run set role ala sudo
Permissionsbullpublic means all usersbullGRANT REVOKE to givetake away privs roles etcbullCONNECT privs on the database (public by default)bullschemas - CREATE USAGE
bullrecommend dropping public or revoke CREATEbullUse per-user or per-app schemas
bulltables - SELECTINSERTUPDATEDELETETRUNCATEbullview - same (incl update) execute as view ownerbullcolumns - SELECTINSERTUPDATEbullfunctions - SECURITY DEFINER are akin to setuid
Default permsbullGenerally secure-by-default
bullExcept functions- EXECUTE granted by defaultbullOwners have all rights on their objectsbullMembership in owning role == ownership
bullALTER DEFAULT PRIVILEGES - for rolesbullFOR ROLE IN SCHEMA GRANTbullCant be applied to just a schema
bullGRANT ON ALL IN SCHEMAbullFor tables views sequences functionsbullOne-time operation new tables will not have privs
TuningbullFor a dedicated serverbullshared_buffers
bullWill be dedicated to PG for cacheingbullUp to half of main memorybullTry 2G on larger servers more may not helpbullPre-93 need to bump sysctl paramsbullPost-93 you dontbullDefaults to 128MB
Tuning (continued)bullwork_mem
bullUsed for in-memory hashing sorts etcbullCan be increased inside a given connectionbullUsed many times over- not a hard limitbullPer connection so be carefulbullDefaults to 1MB (wayy too small)
bullmaintenance_work_membullUsed for building indexesbullMake it larger before building an indexbullDefaults to 16MB (thats a very small index)
Tuning (continued)bulleffective_cache_size
bullTells PG how much of the DB is in memorybullHalf of main memorybullNever allocated only for planning purposesbullDefaults to 128MB
bullautovacuumbullOn a high-rate server make it more aggressivebullIncrease max_workersbullDecrease autovacuum_vacuum_cost_delaybullDefaults are for lightly loaded systems
Config Bump-Upsbullmax_connections = 100
bullConsider using pg_bouncerbull connections == of CPUs is ideal
bullshared_buffers = couple gigbullProbably not more than 3-4G (Test)
bullmaintenance_work_mem = maybe a gigbullUsed for building indexes
bullmax_locks_per_transaction = 128bullMore if you have lots of objectsbull locks available is actually this max_conn
BackupsbullExtremely importantbullpg_basebackup w WAL recieve
bullBinary-based backupbullMUST have WAL files backed up alsobullNeeds to connect to replication DB
bullpg_dumpbullLogical text-based backupbullDoes not back up indexes must rebuildbullRequires lightweight locks on everything
bullTest restoring your data
Monitoringbullcheck_postgresplbullUseful with Nagios Icinga MRTG etcbullProvides metrics as well as monitoringbullAllows custom query for monitoringbullMinimum set of checksarchive_ready (if doing WAL archiving) --- Number of WAL ready filesautovac_freeze --- How close to Autovacuum Max Freezebackends (Metric) --- Number of Backends runningdbstats (Metrics) --- Lots of different statslistener (If using LISTENNOTIFY) --- Checks if anyone is LISTENinglocks (Metric) --- Number of locks heldpgbouncer options (if using pgbouncer) --- Various pgbouncer checkstxn_idle --- Transactions idle for X timetxn_time --- Transactions longer than X timetxn_wraparound --- How close to transaction wraparound
ExtensionsbullInstall -contrib packagebullUse PGXN - httppgxnorgbulltable pg_available_extensions name | default_version | installed_version | comment--------------------+-----------------+-------------------+---------------------------------------------------------------------- file_fdw | 10 | | foreign-data wrapper for flat file access dblink | 10 | | connect to other PostgreSQL databases from within a database plpgsql | 10 | 10 | PLpgSQL procedural language pg_trgm | 10 | | text similarity measurement and index searching based on trigrams adminpack | 10 | | administrative functions for PostgreSQL ip4r | 20 | | hstore | 11 | | data type for storing sets of (key value) pairs
bulladminpack allows superuser to change anythingbulldx lists installed extensions
Thank youStephen Frost
sfrostsnowmannetnet_snow
InstallationbullDebianUbuntuetc
bullaptpostgresqlorgbullAdd PGDG sourceslistd
bullRedHatCentOSetcbullyumpostgresqlorgbullDownload amp Install PGDG RPM
bullMultiple Major Versions
Debian InstallbullConfigs in etcpostgresqlXYmainbullInitial DB in varlibpostgresqlXYmainbullBinaries into usrlibpostgresqlXYbinbullLogs into varlogpostgresqlbullStartup logs in varlogpostgresql alsobullOne init script starts all major versions
Debian ClustersbullDebian provides wrappers and helper scriptsbullpg_lsclusters - lists all PG clustersbullpg_ctlcluster - Control specific clustersbull--cluster option - Specify specific cluster
bullpsql --cluster 92mainbullpg_dump --cluster 92main etc
RedHat InstallbullConfigs in data directorybullDefault DB in varlibpgsqlXYdatabullCreate DB with service postgresql-92 initdbbullBinaries into usrpgsql-XYbinbullLogs into varlibpgsql-XYdatapg_logbullStartup logs in varlibpgsql-XYpgstartuplogbullInit script per major version
PostgreSQL Data DirectorybullSome thing in here do not react well to bulletsbullOn Debian just stay out of itbullOn RedHat be careful to only modify
bullpostgresqlconfbullpg_hbaconfbullpg_identconfbullpg_log
bullDo NOT touch files in pg_xlog or other dirsbullpg_xlog is PGs WAL- not just normal log files
Initial postgresqlconfbulllisten_addresses = (for external access)bullcheckpoint_segments = 30+
bullUses more disk space in pg_xlogbullNever let that partition run out of space
bullcheckpoint_completion_target = 09bullTargets finishing in 90 of time given
bulleffective_cache_size = half the RAMbullNever allocated just for planning
bullmax_wal_senders = 3bullMore later
Loggingbullpostgresqlconf
bulllog_connections = onbulllog_disconnections = onbullline_prefix= m [p] q [l-1] d ur a bulllog_lock_waits = onbulllog_statement = ddlbulllog_min_duration_statement = 100bulllog_temp_files = 0bulllog_autovacuum_min_duration = 0
pg_hbaconfbullControls how users are authenticatedlocal DATABASE USER METHOD [OPTIONS]host DATABASE USER ADDRESS METHOD [OPTIONS]hostssl DATABASE USER ADDRESS METHOD [OPTIONS]hostnossl DATABASE USER ADDRESS METHOD [OPTIONS]
bullRead in order top-to-bottom first match is usedbullhostssl requires SSL connection no is not SSLbullSpecial DBs - all sameuser replicationbullSpecial Users - all + prefix for role membershipbullAddress can be IPv4 or IPv6 can include CIDR maskbullSpecial reject method
Authentication MethodsbullThe ones you should use bullpeer
bullSecure unix-socket-based authbullChecks the Unix username of the user
bullgss (Kerberos)bullIntegreates w MITHeimdal Kerberos and ADbullRecommended for Enterprise deployments
bullcert (SSL Certificate)bullClient-side certificate based authenticationbullUse pg_ident to map CNs to PG usernames
Authentication MethodsbullAcceptable but not idealbullmd5
bullStock usernamepasswordbullUse SSL if youre worried about security
bullpambullModules run as postgres userbullCant be used directly w pam_unixbullsaslauthd can make it work (pam_sasl saslauthd)
bullradiusbullUse SSL if youre worried about security
Auth Method Dontsbulltrust - Never use this- no auth donebullpassword - Password sent in cleartextbullsspi
bullWindows-specificbullUses KerberosGSSAPI underneath
bullidentbullInsecure dont trust it- use peer for local
bullldapbullAuths against an LDAP serverbullUse KerberosGSSAPI if you can
pg_identconfbullDefines mappings which are used in pg_hbamap-name auth-user pg-userkerbnames sfrostSNOWMANNET sfrostcertname stephenfrost sfrost
bullExternal-user to PG-user mappingsbullUnix user joe can be PG user bobbullRegexps can be used- but be carefulbullAlso works for Kerberos client certs etc
Debian configsbullExtra config files in DebianUbuntubullstartconf
bullControls start of this clusterbullCan be auto manual disabled
bullpg_ctlconfbullOptions to pass to pg_ctlbullGenerally dont need to modify it
bullenvironmentbullControlls environment PG starts inbullGenerally dont need to modify it
RedHat configsbullBasically just the initd scripts
Connectingbullsudo su - postgresbullpsqlbull to see backslash-commandsbullh to get help on SQL queriescommandsbullExit with q or ctrl-dbullpsql -h localhost
Looking aroundbulltable pg_stat_activity - aka wbulll - list databases Name | Owner | Encoding | Collate | Ctype | Access privileges-----------+----------+----------+-------------+-------------+----------------------- postgres | postgres | UTF8 | en_USUTF-8 | en_USUTF-8 | template0 | postgres | UTF8 | en_USUTF-8 | en_USUTF-8 | =cpostgres + template1 | postgres | UTF8 | en_USUTF-8 | en_USUTF-8 | =cpostgres +
bulldn - list schemas Name | Owner--------+---------- public | postgres
bulldb - list tablespaces Name | Owner | Location------------+----------+---------- pg_default | postgres | pg_global | postgres |
User setupsbullcreateuser CREATE USERbullpassword to set passwordsbullPrivileges
bullSuperuser- Do not give this outbullCreateRole- Creation and modification of rolesbullCreateDatabase- Allows database creationbullLogin- Allows user to connect to DBbullReplication- Only for replicationsystem userbullAdmin- Allows changing role membershipsbullInherit- Automatically get privileges
RolesbullUsers are really rolesbullGroups are implemented with rolesbullCREATE ROLE (or just createuser --nologin)
bullSame privilege optionsbullCan start as nologin then be granted loginbullCan cascade
bullAny role can be GRANTd to any other rolebullInherit is default acts like group privsbullNoinherit means user must run set role ala sudo
Permissionsbullpublic means all usersbullGRANT REVOKE to givetake away privs roles etcbullCONNECT privs on the database (public by default)bullschemas - CREATE USAGE
bullrecommend dropping public or revoke CREATEbullUse per-user or per-app schemas
bulltables - SELECTINSERTUPDATEDELETETRUNCATEbullview - same (incl update) execute as view ownerbullcolumns - SELECTINSERTUPDATEbullfunctions - SECURITY DEFINER are akin to setuid
Default permsbullGenerally secure-by-default
bullExcept functions- EXECUTE granted by defaultbullOwners have all rights on their objectsbullMembership in owning role == ownership
bullALTER DEFAULT PRIVILEGES - for rolesbullFOR ROLE IN SCHEMA GRANTbullCant be applied to just a schema
bullGRANT ON ALL IN SCHEMAbullFor tables views sequences functionsbullOne-time operation new tables will not have privs
TuningbullFor a dedicated serverbullshared_buffers
bullWill be dedicated to PG for cacheingbullUp to half of main memorybullTry 2G on larger servers more may not helpbullPre-93 need to bump sysctl paramsbullPost-93 you dontbullDefaults to 128MB
Tuning (continued)bullwork_mem
bullUsed for in-memory hashing sorts etcbullCan be increased inside a given connectionbullUsed many times over- not a hard limitbullPer connection so be carefulbullDefaults to 1MB (wayy too small)
bullmaintenance_work_membullUsed for building indexesbullMake it larger before building an indexbullDefaults to 16MB (thats a very small index)
Tuning (continued)bulleffective_cache_size
bullTells PG how much of the DB is in memorybullHalf of main memorybullNever allocated only for planning purposesbullDefaults to 128MB
bullautovacuumbullOn a high-rate server make it more aggressivebullIncrease max_workersbullDecrease autovacuum_vacuum_cost_delaybullDefaults are for lightly loaded systems
Config Bump-Upsbullmax_connections = 100
bullConsider using pg_bouncerbull connections == of CPUs is ideal
bullshared_buffers = couple gigbullProbably not more than 3-4G (Test)
bullmaintenance_work_mem = maybe a gigbullUsed for building indexes
bullmax_locks_per_transaction = 128bullMore if you have lots of objectsbull locks available is actually this max_conn
BackupsbullExtremely importantbullpg_basebackup w WAL recieve
bullBinary-based backupbullMUST have WAL files backed up alsobullNeeds to connect to replication DB
bullpg_dumpbullLogical text-based backupbullDoes not back up indexes must rebuildbullRequires lightweight locks on everything
bullTest restoring your data
Monitoringbullcheck_postgresplbullUseful with Nagios Icinga MRTG etcbullProvides metrics as well as monitoringbullAllows custom query for monitoringbullMinimum set of checksarchive_ready (if doing WAL archiving) --- Number of WAL ready filesautovac_freeze --- How close to Autovacuum Max Freezebackends (Metric) --- Number of Backends runningdbstats (Metrics) --- Lots of different statslistener (If using LISTENNOTIFY) --- Checks if anyone is LISTENinglocks (Metric) --- Number of locks heldpgbouncer options (if using pgbouncer) --- Various pgbouncer checkstxn_idle --- Transactions idle for X timetxn_time --- Transactions longer than X timetxn_wraparound --- How close to transaction wraparound
ExtensionsbullInstall -contrib packagebullUse PGXN - httppgxnorgbulltable pg_available_extensions name | default_version | installed_version | comment--------------------+-----------------+-------------------+---------------------------------------------------------------------- file_fdw | 10 | | foreign-data wrapper for flat file access dblink | 10 | | connect to other PostgreSQL databases from within a database plpgsql | 10 | 10 | PLpgSQL procedural language pg_trgm | 10 | | text similarity measurement and index searching based on trigrams adminpack | 10 | | administrative functions for PostgreSQL ip4r | 20 | | hstore | 11 | | data type for storing sets of (key value) pairs
bulladminpack allows superuser to change anythingbulldx lists installed extensions
Thank youStephen Frost
sfrostsnowmannetnet_snow
Debian InstallbullConfigs in etcpostgresqlXYmainbullInitial DB in varlibpostgresqlXYmainbullBinaries into usrlibpostgresqlXYbinbullLogs into varlogpostgresqlbullStartup logs in varlogpostgresql alsobullOne init script starts all major versions
Debian ClustersbullDebian provides wrappers and helper scriptsbullpg_lsclusters - lists all PG clustersbullpg_ctlcluster - Control specific clustersbull--cluster option - Specify specific cluster
bullpsql --cluster 92mainbullpg_dump --cluster 92main etc
RedHat InstallbullConfigs in data directorybullDefault DB in varlibpgsqlXYdatabullCreate DB with service postgresql-92 initdbbullBinaries into usrpgsql-XYbinbullLogs into varlibpgsql-XYdatapg_logbullStartup logs in varlibpgsql-XYpgstartuplogbullInit script per major version
PostgreSQL Data DirectorybullSome thing in here do not react well to bulletsbullOn Debian just stay out of itbullOn RedHat be careful to only modify
bullpostgresqlconfbullpg_hbaconfbullpg_identconfbullpg_log
bullDo NOT touch files in pg_xlog or other dirsbullpg_xlog is PGs WAL- not just normal log files
Initial postgresqlconfbulllisten_addresses = (for external access)bullcheckpoint_segments = 30+
bullUses more disk space in pg_xlogbullNever let that partition run out of space
bullcheckpoint_completion_target = 09bullTargets finishing in 90 of time given
bulleffective_cache_size = half the RAMbullNever allocated just for planning
bullmax_wal_senders = 3bullMore later
Loggingbullpostgresqlconf
bulllog_connections = onbulllog_disconnections = onbullline_prefix= m [p] q [l-1] d ur a bulllog_lock_waits = onbulllog_statement = ddlbulllog_min_duration_statement = 100bulllog_temp_files = 0bulllog_autovacuum_min_duration = 0
pg_hbaconfbullControls how users are authenticatedlocal DATABASE USER METHOD [OPTIONS]host DATABASE USER ADDRESS METHOD [OPTIONS]hostssl DATABASE USER ADDRESS METHOD [OPTIONS]hostnossl DATABASE USER ADDRESS METHOD [OPTIONS]
bullRead in order top-to-bottom first match is usedbullhostssl requires SSL connection no is not SSLbullSpecial DBs - all sameuser replicationbullSpecial Users - all + prefix for role membershipbullAddress can be IPv4 or IPv6 can include CIDR maskbullSpecial reject method
Authentication MethodsbullThe ones you should use bullpeer
bullSecure unix-socket-based authbullChecks the Unix username of the user
bullgss (Kerberos)bullIntegreates w MITHeimdal Kerberos and ADbullRecommended for Enterprise deployments
bullcert (SSL Certificate)bullClient-side certificate based authenticationbullUse pg_ident to map CNs to PG usernames
Authentication MethodsbullAcceptable but not idealbullmd5
bullStock usernamepasswordbullUse SSL if youre worried about security
bullpambullModules run as postgres userbullCant be used directly w pam_unixbullsaslauthd can make it work (pam_sasl saslauthd)
bullradiusbullUse SSL if youre worried about security
Auth Method Dontsbulltrust - Never use this- no auth donebullpassword - Password sent in cleartextbullsspi
bullWindows-specificbullUses KerberosGSSAPI underneath
bullidentbullInsecure dont trust it- use peer for local
bullldapbullAuths against an LDAP serverbullUse KerberosGSSAPI if you can
pg_identconfbullDefines mappings which are used in pg_hbamap-name auth-user pg-userkerbnames sfrostSNOWMANNET sfrostcertname stephenfrost sfrost
bullExternal-user to PG-user mappingsbullUnix user joe can be PG user bobbullRegexps can be used- but be carefulbullAlso works for Kerberos client certs etc
Debian configsbullExtra config files in DebianUbuntubullstartconf
bullControls start of this clusterbullCan be auto manual disabled
bullpg_ctlconfbullOptions to pass to pg_ctlbullGenerally dont need to modify it
bullenvironmentbullControlls environment PG starts inbullGenerally dont need to modify it
RedHat configsbullBasically just the initd scripts
Connectingbullsudo su - postgresbullpsqlbull to see backslash-commandsbullh to get help on SQL queriescommandsbullExit with q or ctrl-dbullpsql -h localhost
Looking aroundbulltable pg_stat_activity - aka wbulll - list databases Name | Owner | Encoding | Collate | Ctype | Access privileges-----------+----------+----------+-------------+-------------+----------------------- postgres | postgres | UTF8 | en_USUTF-8 | en_USUTF-8 | template0 | postgres | UTF8 | en_USUTF-8 | en_USUTF-8 | =cpostgres + template1 | postgres | UTF8 | en_USUTF-8 | en_USUTF-8 | =cpostgres +
bulldn - list schemas Name | Owner--------+---------- public | postgres
bulldb - list tablespaces Name | Owner | Location------------+----------+---------- pg_default | postgres | pg_global | postgres |
User setupsbullcreateuser CREATE USERbullpassword to set passwordsbullPrivileges
bullSuperuser- Do not give this outbullCreateRole- Creation and modification of rolesbullCreateDatabase- Allows database creationbullLogin- Allows user to connect to DBbullReplication- Only for replicationsystem userbullAdmin- Allows changing role membershipsbullInherit- Automatically get privileges
RolesbullUsers are really rolesbullGroups are implemented with rolesbullCREATE ROLE (or just createuser --nologin)
bullSame privilege optionsbullCan start as nologin then be granted loginbullCan cascade
bullAny role can be GRANTd to any other rolebullInherit is default acts like group privsbullNoinherit means user must run set role ala sudo
Permissionsbullpublic means all usersbullGRANT REVOKE to givetake away privs roles etcbullCONNECT privs on the database (public by default)bullschemas - CREATE USAGE
bullrecommend dropping public or revoke CREATEbullUse per-user or per-app schemas
bulltables - SELECTINSERTUPDATEDELETETRUNCATEbullview - same (incl update) execute as view ownerbullcolumns - SELECTINSERTUPDATEbullfunctions - SECURITY DEFINER are akin to setuid
Default permsbullGenerally secure-by-default
bullExcept functions- EXECUTE granted by defaultbullOwners have all rights on their objectsbullMembership in owning role == ownership
bullALTER DEFAULT PRIVILEGES - for rolesbullFOR ROLE IN SCHEMA GRANTbullCant be applied to just a schema
bullGRANT ON ALL IN SCHEMAbullFor tables views sequences functionsbullOne-time operation new tables will not have privs
TuningbullFor a dedicated serverbullshared_buffers
bullWill be dedicated to PG for cacheingbullUp to half of main memorybullTry 2G on larger servers more may not helpbullPre-93 need to bump sysctl paramsbullPost-93 you dontbullDefaults to 128MB
Tuning (continued)bullwork_mem
bullUsed for in-memory hashing sorts etcbullCan be increased inside a given connectionbullUsed many times over- not a hard limitbullPer connection so be carefulbullDefaults to 1MB (wayy too small)
bullmaintenance_work_membullUsed for building indexesbullMake it larger before building an indexbullDefaults to 16MB (thats a very small index)
Tuning (continued)bulleffective_cache_size
bullTells PG how much of the DB is in memorybullHalf of main memorybullNever allocated only for planning purposesbullDefaults to 128MB
bullautovacuumbullOn a high-rate server make it more aggressivebullIncrease max_workersbullDecrease autovacuum_vacuum_cost_delaybullDefaults are for lightly loaded systems
Config Bump-Upsbullmax_connections = 100
bullConsider using pg_bouncerbull connections == of CPUs is ideal
bullshared_buffers = couple gigbullProbably not more than 3-4G (Test)
bullmaintenance_work_mem = maybe a gigbullUsed for building indexes
bullmax_locks_per_transaction = 128bullMore if you have lots of objectsbull locks available is actually this max_conn
BackupsbullExtremely importantbullpg_basebackup w WAL recieve
bullBinary-based backupbullMUST have WAL files backed up alsobullNeeds to connect to replication DB
bullpg_dumpbullLogical text-based backupbullDoes not back up indexes must rebuildbullRequires lightweight locks on everything
bullTest restoring your data
Monitoringbullcheck_postgresplbullUseful with Nagios Icinga MRTG etcbullProvides metrics as well as monitoringbullAllows custom query for monitoringbullMinimum set of checksarchive_ready (if doing WAL archiving) --- Number of WAL ready filesautovac_freeze --- How close to Autovacuum Max Freezebackends (Metric) --- Number of Backends runningdbstats (Metrics) --- Lots of different statslistener (If using LISTENNOTIFY) --- Checks if anyone is LISTENinglocks (Metric) --- Number of locks heldpgbouncer options (if using pgbouncer) --- Various pgbouncer checkstxn_idle --- Transactions idle for X timetxn_time --- Transactions longer than X timetxn_wraparound --- How close to transaction wraparound
ExtensionsbullInstall -contrib packagebullUse PGXN - httppgxnorgbulltable pg_available_extensions name | default_version | installed_version | comment--------------------+-----------------+-------------------+---------------------------------------------------------------------- file_fdw | 10 | | foreign-data wrapper for flat file access dblink | 10 | | connect to other PostgreSQL databases from within a database plpgsql | 10 | 10 | PLpgSQL procedural language pg_trgm | 10 | | text similarity measurement and index searching based on trigrams adminpack | 10 | | administrative functions for PostgreSQL ip4r | 20 | | hstore | 11 | | data type for storing sets of (key value) pairs
bulladminpack allows superuser to change anythingbulldx lists installed extensions
Thank youStephen Frost
sfrostsnowmannetnet_snow
Debian ClustersbullDebian provides wrappers and helper scriptsbullpg_lsclusters - lists all PG clustersbullpg_ctlcluster - Control specific clustersbull--cluster option - Specify specific cluster
bullpsql --cluster 92mainbullpg_dump --cluster 92main etc
RedHat InstallbullConfigs in data directorybullDefault DB in varlibpgsqlXYdatabullCreate DB with service postgresql-92 initdbbullBinaries into usrpgsql-XYbinbullLogs into varlibpgsql-XYdatapg_logbullStartup logs in varlibpgsql-XYpgstartuplogbullInit script per major version
PostgreSQL Data DirectorybullSome thing in here do not react well to bulletsbullOn Debian just stay out of itbullOn RedHat be careful to only modify
bullpostgresqlconfbullpg_hbaconfbullpg_identconfbullpg_log
bullDo NOT touch files in pg_xlog or other dirsbullpg_xlog is PGs WAL- not just normal log files
Initial postgresqlconfbulllisten_addresses = (for external access)bullcheckpoint_segments = 30+
bullUses more disk space in pg_xlogbullNever let that partition run out of space
bullcheckpoint_completion_target = 09bullTargets finishing in 90 of time given
bulleffective_cache_size = half the RAMbullNever allocated just for planning
bullmax_wal_senders = 3bullMore later
Loggingbullpostgresqlconf
bulllog_connections = onbulllog_disconnections = onbullline_prefix= m [p] q [l-1] d ur a bulllog_lock_waits = onbulllog_statement = ddlbulllog_min_duration_statement = 100bulllog_temp_files = 0bulllog_autovacuum_min_duration = 0
pg_hbaconfbullControls how users are authenticatedlocal DATABASE USER METHOD [OPTIONS]host DATABASE USER ADDRESS METHOD [OPTIONS]hostssl DATABASE USER ADDRESS METHOD [OPTIONS]hostnossl DATABASE USER ADDRESS METHOD [OPTIONS]
bullRead in order top-to-bottom first match is usedbullhostssl requires SSL connection no is not SSLbullSpecial DBs - all sameuser replicationbullSpecial Users - all + prefix for role membershipbullAddress can be IPv4 or IPv6 can include CIDR maskbullSpecial reject method
Authentication MethodsbullThe ones you should use bullpeer
bullSecure unix-socket-based authbullChecks the Unix username of the user
bullgss (Kerberos)bullIntegreates w MITHeimdal Kerberos and ADbullRecommended for Enterprise deployments
bullcert (SSL Certificate)bullClient-side certificate based authenticationbullUse pg_ident to map CNs to PG usernames
Authentication MethodsbullAcceptable but not idealbullmd5
bullStock usernamepasswordbullUse SSL if youre worried about security
bullpambullModules run as postgres userbullCant be used directly w pam_unixbullsaslauthd can make it work (pam_sasl saslauthd)
bullradiusbullUse SSL if youre worried about security
Auth Method Dontsbulltrust - Never use this- no auth donebullpassword - Password sent in cleartextbullsspi
bullWindows-specificbullUses KerberosGSSAPI underneath
bullidentbullInsecure dont trust it- use peer for local
bullldapbullAuths against an LDAP serverbullUse KerberosGSSAPI if you can
pg_identconfbullDefines mappings which are used in pg_hbamap-name auth-user pg-userkerbnames sfrostSNOWMANNET sfrostcertname stephenfrost sfrost
bullExternal-user to PG-user mappingsbullUnix user joe can be PG user bobbullRegexps can be used- but be carefulbullAlso works for Kerberos client certs etc
Debian configsbullExtra config files in DebianUbuntubullstartconf
bullControls start of this clusterbullCan be auto manual disabled
bullpg_ctlconfbullOptions to pass to pg_ctlbullGenerally dont need to modify it
bullenvironmentbullControlls environment PG starts inbullGenerally dont need to modify it
RedHat configsbullBasically just the initd scripts
Connectingbullsudo su - postgresbullpsqlbull to see backslash-commandsbullh to get help on SQL queriescommandsbullExit with q or ctrl-dbullpsql -h localhost
Looking aroundbulltable pg_stat_activity - aka wbulll - list databases Name | Owner | Encoding | Collate | Ctype | Access privileges-----------+----------+----------+-------------+-------------+----------------------- postgres | postgres | UTF8 | en_USUTF-8 | en_USUTF-8 | template0 | postgres | UTF8 | en_USUTF-8 | en_USUTF-8 | =cpostgres + template1 | postgres | UTF8 | en_USUTF-8 | en_USUTF-8 | =cpostgres +
bulldn - list schemas Name | Owner--------+---------- public | postgres
bulldb - list tablespaces Name | Owner | Location------------+----------+---------- pg_default | postgres | pg_global | postgres |
User setupsbullcreateuser CREATE USERbullpassword to set passwordsbullPrivileges
bullSuperuser- Do not give this outbullCreateRole- Creation and modification of rolesbullCreateDatabase- Allows database creationbullLogin- Allows user to connect to DBbullReplication- Only for replicationsystem userbullAdmin- Allows changing role membershipsbullInherit- Automatically get privileges
RolesbullUsers are really rolesbullGroups are implemented with rolesbullCREATE ROLE (or just createuser --nologin)
bullSame privilege optionsbullCan start as nologin then be granted loginbullCan cascade
bullAny role can be GRANTd to any other rolebullInherit is default acts like group privsbullNoinherit means user must run set role ala sudo
Permissionsbullpublic means all usersbullGRANT REVOKE to givetake away privs roles etcbullCONNECT privs on the database (public by default)bullschemas - CREATE USAGE
bullrecommend dropping public or revoke CREATEbullUse per-user or per-app schemas
bulltables - SELECTINSERTUPDATEDELETETRUNCATEbullview - same (incl update) execute as view ownerbullcolumns - SELECTINSERTUPDATEbullfunctions - SECURITY DEFINER are akin to setuid
Default permsbullGenerally secure-by-default
bullExcept functions- EXECUTE granted by defaultbullOwners have all rights on their objectsbullMembership in owning role == ownership
bullALTER DEFAULT PRIVILEGES - for rolesbullFOR ROLE IN SCHEMA GRANTbullCant be applied to just a schema
bullGRANT ON ALL IN SCHEMAbullFor tables views sequences functionsbullOne-time operation new tables will not have privs
TuningbullFor a dedicated serverbullshared_buffers
bullWill be dedicated to PG for cacheingbullUp to half of main memorybullTry 2G on larger servers more may not helpbullPre-93 need to bump sysctl paramsbullPost-93 you dontbullDefaults to 128MB
Tuning (continued)bullwork_mem
bullUsed for in-memory hashing sorts etcbullCan be increased inside a given connectionbullUsed many times over- not a hard limitbullPer connection so be carefulbullDefaults to 1MB (wayy too small)
bullmaintenance_work_membullUsed for building indexesbullMake it larger before building an indexbullDefaults to 16MB (thats a very small index)
Tuning (continued)bulleffective_cache_size
bullTells PG how much of the DB is in memorybullHalf of main memorybullNever allocated only for planning purposesbullDefaults to 128MB
bullautovacuumbullOn a high-rate server make it more aggressivebullIncrease max_workersbullDecrease autovacuum_vacuum_cost_delaybullDefaults are for lightly loaded systems
Config Bump-Upsbullmax_connections = 100
bullConsider using pg_bouncerbull connections == of CPUs is ideal
bullshared_buffers = couple gigbullProbably not more than 3-4G (Test)
bullmaintenance_work_mem = maybe a gigbullUsed for building indexes
bullmax_locks_per_transaction = 128bullMore if you have lots of objectsbull locks available is actually this max_conn
BackupsbullExtremely importantbullpg_basebackup w WAL recieve
bullBinary-based backupbullMUST have WAL files backed up alsobullNeeds to connect to replication DB
bullpg_dumpbullLogical text-based backupbullDoes not back up indexes must rebuildbullRequires lightweight locks on everything
bullTest restoring your data
Monitoringbullcheck_postgresplbullUseful with Nagios Icinga MRTG etcbullProvides metrics as well as monitoringbullAllows custom query for monitoringbullMinimum set of checksarchive_ready (if doing WAL archiving) --- Number of WAL ready filesautovac_freeze --- How close to Autovacuum Max Freezebackends (Metric) --- Number of Backends runningdbstats (Metrics) --- Lots of different statslistener (If using LISTENNOTIFY) --- Checks if anyone is LISTENinglocks (Metric) --- Number of locks heldpgbouncer options (if using pgbouncer) --- Various pgbouncer checkstxn_idle --- Transactions idle for X timetxn_time --- Transactions longer than X timetxn_wraparound --- How close to transaction wraparound
ExtensionsbullInstall -contrib packagebullUse PGXN - httppgxnorgbulltable pg_available_extensions name | default_version | installed_version | comment--------------------+-----------------+-------------------+---------------------------------------------------------------------- file_fdw | 10 | | foreign-data wrapper for flat file access dblink | 10 | | connect to other PostgreSQL databases from within a database plpgsql | 10 | 10 | PLpgSQL procedural language pg_trgm | 10 | | text similarity measurement and index searching based on trigrams adminpack | 10 | | administrative functions for PostgreSQL ip4r | 20 | | hstore | 11 | | data type for storing sets of (key value) pairs
bulladminpack allows superuser to change anythingbulldx lists installed extensions
Thank youStephen Frost
sfrostsnowmannetnet_snow
RedHat InstallbullConfigs in data directorybullDefault DB in varlibpgsqlXYdatabullCreate DB with service postgresql-92 initdbbullBinaries into usrpgsql-XYbinbullLogs into varlibpgsql-XYdatapg_logbullStartup logs in varlibpgsql-XYpgstartuplogbullInit script per major version
PostgreSQL Data DirectorybullSome thing in here do not react well to bulletsbullOn Debian just stay out of itbullOn RedHat be careful to only modify
bullpostgresqlconfbullpg_hbaconfbullpg_identconfbullpg_log
bullDo NOT touch files in pg_xlog or other dirsbullpg_xlog is PGs WAL- not just normal log files
Initial postgresqlconfbulllisten_addresses = (for external access)bullcheckpoint_segments = 30+
bullUses more disk space in pg_xlogbullNever let that partition run out of space
bullcheckpoint_completion_target = 09bullTargets finishing in 90 of time given
bulleffective_cache_size = half the RAMbullNever allocated just for planning
bullmax_wal_senders = 3bullMore later
Loggingbullpostgresqlconf
bulllog_connections = onbulllog_disconnections = onbullline_prefix= m [p] q [l-1] d ur a bulllog_lock_waits = onbulllog_statement = ddlbulllog_min_duration_statement = 100bulllog_temp_files = 0bulllog_autovacuum_min_duration = 0
pg_hbaconfbullControls how users are authenticatedlocal DATABASE USER METHOD [OPTIONS]host DATABASE USER ADDRESS METHOD [OPTIONS]hostssl DATABASE USER ADDRESS METHOD [OPTIONS]hostnossl DATABASE USER ADDRESS METHOD [OPTIONS]
bullRead in order top-to-bottom first match is usedbullhostssl requires SSL connection no is not SSLbullSpecial DBs - all sameuser replicationbullSpecial Users - all + prefix for role membershipbullAddress can be IPv4 or IPv6 can include CIDR maskbullSpecial reject method
Authentication MethodsbullThe ones you should use bullpeer
bullSecure unix-socket-based authbullChecks the Unix username of the user
bullgss (Kerberos)bullIntegreates w MITHeimdal Kerberos and ADbullRecommended for Enterprise deployments
bullcert (SSL Certificate)bullClient-side certificate based authenticationbullUse pg_ident to map CNs to PG usernames
Authentication MethodsbullAcceptable but not idealbullmd5
bullStock usernamepasswordbullUse SSL if youre worried about security
bullpambullModules run as postgres userbullCant be used directly w pam_unixbullsaslauthd can make it work (pam_sasl saslauthd)
bullradiusbullUse SSL if youre worried about security
Auth Method Dontsbulltrust - Never use this- no auth donebullpassword - Password sent in cleartextbullsspi
bullWindows-specificbullUses KerberosGSSAPI underneath
bullidentbullInsecure dont trust it- use peer for local
bullldapbullAuths against an LDAP serverbullUse KerberosGSSAPI if you can
pg_identconfbullDefines mappings which are used in pg_hbamap-name auth-user pg-userkerbnames sfrostSNOWMANNET sfrostcertname stephenfrost sfrost
bullExternal-user to PG-user mappingsbullUnix user joe can be PG user bobbullRegexps can be used- but be carefulbullAlso works for Kerberos client certs etc
Debian configsbullExtra config files in DebianUbuntubullstartconf
bullControls start of this clusterbullCan be auto manual disabled
bullpg_ctlconfbullOptions to pass to pg_ctlbullGenerally dont need to modify it
bullenvironmentbullControlls environment PG starts inbullGenerally dont need to modify it
RedHat configsbullBasically just the initd scripts
Connectingbullsudo su - postgresbullpsqlbull to see backslash-commandsbullh to get help on SQL queriescommandsbullExit with q or ctrl-dbullpsql -h localhost
Looking aroundbulltable pg_stat_activity - aka wbulll - list databases Name | Owner | Encoding | Collate | Ctype | Access privileges-----------+----------+----------+-------------+-------------+----------------------- postgres | postgres | UTF8 | en_USUTF-8 | en_USUTF-8 | template0 | postgres | UTF8 | en_USUTF-8 | en_USUTF-8 | =cpostgres + template1 | postgres | UTF8 | en_USUTF-8 | en_USUTF-8 | =cpostgres +
bulldn - list schemas Name | Owner--------+---------- public | postgres
bulldb - list tablespaces Name | Owner | Location------------+----------+---------- pg_default | postgres | pg_global | postgres |
User setupsbullcreateuser CREATE USERbullpassword to set passwordsbullPrivileges
bullSuperuser- Do not give this outbullCreateRole- Creation and modification of rolesbullCreateDatabase- Allows database creationbullLogin- Allows user to connect to DBbullReplication- Only for replicationsystem userbullAdmin- Allows changing role membershipsbullInherit- Automatically get privileges
RolesbullUsers are really rolesbullGroups are implemented with rolesbullCREATE ROLE (or just createuser --nologin)
bullSame privilege optionsbullCan start as nologin then be granted loginbullCan cascade
bullAny role can be GRANTd to any other rolebullInherit is default acts like group privsbullNoinherit means user must run set role ala sudo
Permissionsbullpublic means all usersbullGRANT REVOKE to givetake away privs roles etcbullCONNECT privs on the database (public by default)bullschemas - CREATE USAGE
bullrecommend dropping public or revoke CREATEbullUse per-user or per-app schemas
bulltables - SELECTINSERTUPDATEDELETETRUNCATEbullview - same (incl update) execute as view ownerbullcolumns - SELECTINSERTUPDATEbullfunctions - SECURITY DEFINER are akin to setuid
Default permsbullGenerally secure-by-default
bullExcept functions- EXECUTE granted by defaultbullOwners have all rights on their objectsbullMembership in owning role == ownership
bullALTER DEFAULT PRIVILEGES - for rolesbullFOR ROLE IN SCHEMA GRANTbullCant be applied to just a schema
bullGRANT ON ALL IN SCHEMAbullFor tables views sequences functionsbullOne-time operation new tables will not have privs
TuningbullFor a dedicated serverbullshared_buffers
bullWill be dedicated to PG for cacheingbullUp to half of main memorybullTry 2G on larger servers more may not helpbullPre-93 need to bump sysctl paramsbullPost-93 you dontbullDefaults to 128MB
Tuning (continued)bullwork_mem
bullUsed for in-memory hashing sorts etcbullCan be increased inside a given connectionbullUsed many times over- not a hard limitbullPer connection so be carefulbullDefaults to 1MB (wayy too small)
bullmaintenance_work_membullUsed for building indexesbullMake it larger before building an indexbullDefaults to 16MB (thats a very small index)
Tuning (continued)bulleffective_cache_size
bullTells PG how much of the DB is in memorybullHalf of main memorybullNever allocated only for planning purposesbullDefaults to 128MB
bullautovacuumbullOn a high-rate server make it more aggressivebullIncrease max_workersbullDecrease autovacuum_vacuum_cost_delaybullDefaults are for lightly loaded systems
Config Bump-Upsbullmax_connections = 100
bullConsider using pg_bouncerbull connections == of CPUs is ideal
bullshared_buffers = couple gigbullProbably not more than 3-4G (Test)
bullmaintenance_work_mem = maybe a gigbullUsed for building indexes
bullmax_locks_per_transaction = 128bullMore if you have lots of objectsbull locks available is actually this max_conn
BackupsbullExtremely importantbullpg_basebackup w WAL recieve
bullBinary-based backupbullMUST have WAL files backed up alsobullNeeds to connect to replication DB
bullpg_dumpbullLogical text-based backupbullDoes not back up indexes must rebuildbullRequires lightweight locks on everything
bullTest restoring your data
Monitoringbullcheck_postgresplbullUseful with Nagios Icinga MRTG etcbullProvides metrics as well as monitoringbullAllows custom query for monitoringbullMinimum set of checksarchive_ready (if doing WAL archiving) --- Number of WAL ready filesautovac_freeze --- How close to Autovacuum Max Freezebackends (Metric) --- Number of Backends runningdbstats (Metrics) --- Lots of different statslistener (If using LISTENNOTIFY) --- Checks if anyone is LISTENinglocks (Metric) --- Number of locks heldpgbouncer options (if using pgbouncer) --- Various pgbouncer checkstxn_idle --- Transactions idle for X timetxn_time --- Transactions longer than X timetxn_wraparound --- How close to transaction wraparound
ExtensionsbullInstall -contrib packagebullUse PGXN - httppgxnorgbulltable pg_available_extensions name | default_version | installed_version | comment--------------------+-----------------+-------------------+---------------------------------------------------------------------- file_fdw | 10 | | foreign-data wrapper for flat file access dblink | 10 | | connect to other PostgreSQL databases from within a database plpgsql | 10 | 10 | PLpgSQL procedural language pg_trgm | 10 | | text similarity measurement and index searching based on trigrams adminpack | 10 | | administrative functions for PostgreSQL ip4r | 20 | | hstore | 11 | | data type for storing sets of (key value) pairs
bulladminpack allows superuser to change anythingbulldx lists installed extensions
Thank youStephen Frost
sfrostsnowmannetnet_snow
PostgreSQL Data DirectorybullSome thing in here do not react well to bulletsbullOn Debian just stay out of itbullOn RedHat be careful to only modify
bullpostgresqlconfbullpg_hbaconfbullpg_identconfbullpg_log
bullDo NOT touch files in pg_xlog or other dirsbullpg_xlog is PGs WAL- not just normal log files
Initial postgresqlconfbulllisten_addresses = (for external access)bullcheckpoint_segments = 30+
bullUses more disk space in pg_xlogbullNever let that partition run out of space
bullcheckpoint_completion_target = 09bullTargets finishing in 90 of time given
bulleffective_cache_size = half the RAMbullNever allocated just for planning
bullmax_wal_senders = 3bullMore later
Loggingbullpostgresqlconf
bulllog_connections = onbulllog_disconnections = onbullline_prefix= m [p] q [l-1] d ur a bulllog_lock_waits = onbulllog_statement = ddlbulllog_min_duration_statement = 100bulllog_temp_files = 0bulllog_autovacuum_min_duration = 0
pg_hbaconfbullControls how users are authenticatedlocal DATABASE USER METHOD [OPTIONS]host DATABASE USER ADDRESS METHOD [OPTIONS]hostssl DATABASE USER ADDRESS METHOD [OPTIONS]hostnossl DATABASE USER ADDRESS METHOD [OPTIONS]
bullRead in order top-to-bottom first match is usedbullhostssl requires SSL connection no is not SSLbullSpecial DBs - all sameuser replicationbullSpecial Users - all + prefix for role membershipbullAddress can be IPv4 or IPv6 can include CIDR maskbullSpecial reject method
Authentication MethodsbullThe ones you should use bullpeer
bullSecure unix-socket-based authbullChecks the Unix username of the user
bullgss (Kerberos)bullIntegreates w MITHeimdal Kerberos and ADbullRecommended for Enterprise deployments
bullcert (SSL Certificate)bullClient-side certificate based authenticationbullUse pg_ident to map CNs to PG usernames
Authentication MethodsbullAcceptable but not idealbullmd5
bullStock usernamepasswordbullUse SSL if youre worried about security
bullpambullModules run as postgres userbullCant be used directly w pam_unixbullsaslauthd can make it work (pam_sasl saslauthd)
bullradiusbullUse SSL if youre worried about security
Auth Method Dontsbulltrust - Never use this- no auth donebullpassword - Password sent in cleartextbullsspi
bullWindows-specificbullUses KerberosGSSAPI underneath
bullidentbullInsecure dont trust it- use peer for local
bullldapbullAuths against an LDAP serverbullUse KerberosGSSAPI if you can
pg_identconfbullDefines mappings which are used in pg_hbamap-name auth-user pg-userkerbnames sfrostSNOWMANNET sfrostcertname stephenfrost sfrost
bullExternal-user to PG-user mappingsbullUnix user joe can be PG user bobbullRegexps can be used- but be carefulbullAlso works for Kerberos client certs etc
Debian configsbullExtra config files in DebianUbuntubullstartconf
bullControls start of this clusterbullCan be auto manual disabled
bullpg_ctlconfbullOptions to pass to pg_ctlbullGenerally dont need to modify it
bullenvironmentbullControlls environment PG starts inbullGenerally dont need to modify it
RedHat configsbullBasically just the initd scripts
Connectingbullsudo su - postgresbullpsqlbull to see backslash-commandsbullh to get help on SQL queriescommandsbullExit with q or ctrl-dbullpsql -h localhost
Looking aroundbulltable pg_stat_activity - aka wbulll - list databases Name | Owner | Encoding | Collate | Ctype | Access privileges-----------+----------+----------+-------------+-------------+----------------------- postgres | postgres | UTF8 | en_USUTF-8 | en_USUTF-8 | template0 | postgres | UTF8 | en_USUTF-8 | en_USUTF-8 | =cpostgres + template1 | postgres | UTF8 | en_USUTF-8 | en_USUTF-8 | =cpostgres +
bulldn - list schemas Name | Owner--------+---------- public | postgres
bulldb - list tablespaces Name | Owner | Location------------+----------+---------- pg_default | postgres | pg_global | postgres |
User setupsbullcreateuser CREATE USERbullpassword to set passwordsbullPrivileges
bullSuperuser- Do not give this outbullCreateRole- Creation and modification of rolesbullCreateDatabase- Allows database creationbullLogin- Allows user to connect to DBbullReplication- Only for replicationsystem userbullAdmin- Allows changing role membershipsbullInherit- Automatically get privileges
RolesbullUsers are really rolesbullGroups are implemented with rolesbullCREATE ROLE (or just createuser --nologin)
bullSame privilege optionsbullCan start as nologin then be granted loginbullCan cascade
bullAny role can be GRANTd to any other rolebullInherit is default acts like group privsbullNoinherit means user must run set role ala sudo
Permissionsbullpublic means all usersbullGRANT REVOKE to givetake away privs roles etcbullCONNECT privs on the database (public by default)bullschemas - CREATE USAGE
bullrecommend dropping public or revoke CREATEbullUse per-user or per-app schemas
bulltables - SELECTINSERTUPDATEDELETETRUNCATEbullview - same (incl update) execute as view ownerbullcolumns - SELECTINSERTUPDATEbullfunctions - SECURITY DEFINER are akin to setuid
Default permsbullGenerally secure-by-default
bullExcept functions- EXECUTE granted by defaultbullOwners have all rights on their objectsbullMembership in owning role == ownership
bullALTER DEFAULT PRIVILEGES - for rolesbullFOR ROLE IN SCHEMA GRANTbullCant be applied to just a schema
bullGRANT ON ALL IN SCHEMAbullFor tables views sequences functionsbullOne-time operation new tables will not have privs
TuningbullFor a dedicated serverbullshared_buffers
bullWill be dedicated to PG for cacheingbullUp to half of main memorybullTry 2G on larger servers more may not helpbullPre-93 need to bump sysctl paramsbullPost-93 you dontbullDefaults to 128MB
Tuning (continued)bullwork_mem
bullUsed for in-memory hashing sorts etcbullCan be increased inside a given connectionbullUsed many times over- not a hard limitbullPer connection so be carefulbullDefaults to 1MB (wayy too small)
bullmaintenance_work_membullUsed for building indexesbullMake it larger before building an indexbullDefaults to 16MB (thats a very small index)
Tuning (continued)bulleffective_cache_size
bullTells PG how much of the DB is in memorybullHalf of main memorybullNever allocated only for planning purposesbullDefaults to 128MB
bullautovacuumbullOn a high-rate server make it more aggressivebullIncrease max_workersbullDecrease autovacuum_vacuum_cost_delaybullDefaults are for lightly loaded systems
Config Bump-Upsbullmax_connections = 100
bullConsider using pg_bouncerbull connections == of CPUs is ideal
bullshared_buffers = couple gigbullProbably not more than 3-4G (Test)
bullmaintenance_work_mem = maybe a gigbullUsed for building indexes
bullmax_locks_per_transaction = 128bullMore if you have lots of objectsbull locks available is actually this max_conn
BackupsbullExtremely importantbullpg_basebackup w WAL recieve
bullBinary-based backupbullMUST have WAL files backed up alsobullNeeds to connect to replication DB
bullpg_dumpbullLogical text-based backupbullDoes not back up indexes must rebuildbullRequires lightweight locks on everything
bullTest restoring your data
Monitoringbullcheck_postgresplbullUseful with Nagios Icinga MRTG etcbullProvides metrics as well as monitoringbullAllows custom query for monitoringbullMinimum set of checksarchive_ready (if doing WAL archiving) --- Number of WAL ready filesautovac_freeze --- How close to Autovacuum Max Freezebackends (Metric) --- Number of Backends runningdbstats (Metrics) --- Lots of different statslistener (If using LISTENNOTIFY) --- Checks if anyone is LISTENinglocks (Metric) --- Number of locks heldpgbouncer options (if using pgbouncer) --- Various pgbouncer checkstxn_idle --- Transactions idle for X timetxn_time --- Transactions longer than X timetxn_wraparound --- How close to transaction wraparound
ExtensionsbullInstall -contrib packagebullUse PGXN - httppgxnorgbulltable pg_available_extensions name | default_version | installed_version | comment--------------------+-----------------+-------------------+---------------------------------------------------------------------- file_fdw | 10 | | foreign-data wrapper for flat file access dblink | 10 | | connect to other PostgreSQL databases from within a database plpgsql | 10 | 10 | PLpgSQL procedural language pg_trgm | 10 | | text similarity measurement and index searching based on trigrams adminpack | 10 | | administrative functions for PostgreSQL ip4r | 20 | | hstore | 11 | | data type for storing sets of (key value) pairs
bulladminpack allows superuser to change anythingbulldx lists installed extensions
Thank youStephen Frost
sfrostsnowmannetnet_snow
Initial postgresqlconfbulllisten_addresses = (for external access)bullcheckpoint_segments = 30+
bullUses more disk space in pg_xlogbullNever let that partition run out of space
bullcheckpoint_completion_target = 09bullTargets finishing in 90 of time given
bulleffective_cache_size = half the RAMbullNever allocated just for planning
bullmax_wal_senders = 3bullMore later
Loggingbullpostgresqlconf
bulllog_connections = onbulllog_disconnections = onbullline_prefix= m [p] q [l-1] d ur a bulllog_lock_waits = onbulllog_statement = ddlbulllog_min_duration_statement = 100bulllog_temp_files = 0bulllog_autovacuum_min_duration = 0
pg_hbaconfbullControls how users are authenticatedlocal DATABASE USER METHOD [OPTIONS]host DATABASE USER ADDRESS METHOD [OPTIONS]hostssl DATABASE USER ADDRESS METHOD [OPTIONS]hostnossl DATABASE USER ADDRESS METHOD [OPTIONS]
bullRead in order top-to-bottom first match is usedbullhostssl requires SSL connection no is not SSLbullSpecial DBs - all sameuser replicationbullSpecial Users - all + prefix for role membershipbullAddress can be IPv4 or IPv6 can include CIDR maskbullSpecial reject method
Authentication MethodsbullThe ones you should use bullpeer
bullSecure unix-socket-based authbullChecks the Unix username of the user
bullgss (Kerberos)bullIntegreates w MITHeimdal Kerberos and ADbullRecommended for Enterprise deployments
bullcert (SSL Certificate)bullClient-side certificate based authenticationbullUse pg_ident to map CNs to PG usernames
Authentication MethodsbullAcceptable but not idealbullmd5
bullStock usernamepasswordbullUse SSL if youre worried about security
bullpambullModules run as postgres userbullCant be used directly w pam_unixbullsaslauthd can make it work (pam_sasl saslauthd)
bullradiusbullUse SSL if youre worried about security
Auth Method Dontsbulltrust - Never use this- no auth donebullpassword - Password sent in cleartextbullsspi
bullWindows-specificbullUses KerberosGSSAPI underneath
bullidentbullInsecure dont trust it- use peer for local
bullldapbullAuths against an LDAP serverbullUse KerberosGSSAPI if you can
pg_identconfbullDefines mappings which are used in pg_hbamap-name auth-user pg-userkerbnames sfrostSNOWMANNET sfrostcertname stephenfrost sfrost
bullExternal-user to PG-user mappingsbullUnix user joe can be PG user bobbullRegexps can be used- but be carefulbullAlso works for Kerberos client certs etc
Debian configsbullExtra config files in DebianUbuntubullstartconf
bullControls start of this clusterbullCan be auto manual disabled
bullpg_ctlconfbullOptions to pass to pg_ctlbullGenerally dont need to modify it
bullenvironmentbullControlls environment PG starts inbullGenerally dont need to modify it
RedHat configsbullBasically just the initd scripts
Connectingbullsudo su - postgresbullpsqlbull to see backslash-commandsbullh to get help on SQL queriescommandsbullExit with q or ctrl-dbullpsql -h localhost
Looking aroundbulltable pg_stat_activity - aka wbulll - list databases Name | Owner | Encoding | Collate | Ctype | Access privileges-----------+----------+----------+-------------+-------------+----------------------- postgres | postgres | UTF8 | en_USUTF-8 | en_USUTF-8 | template0 | postgres | UTF8 | en_USUTF-8 | en_USUTF-8 | =cpostgres + template1 | postgres | UTF8 | en_USUTF-8 | en_USUTF-8 | =cpostgres +
bulldn - list schemas Name | Owner--------+---------- public | postgres
bulldb - list tablespaces Name | Owner | Location------------+----------+---------- pg_default | postgres | pg_global | postgres |
User setupsbullcreateuser CREATE USERbullpassword to set passwordsbullPrivileges
bullSuperuser- Do not give this outbullCreateRole- Creation and modification of rolesbullCreateDatabase- Allows database creationbullLogin- Allows user to connect to DBbullReplication- Only for replicationsystem userbullAdmin- Allows changing role membershipsbullInherit- Automatically get privileges
RolesbullUsers are really rolesbullGroups are implemented with rolesbullCREATE ROLE (or just createuser --nologin)
bullSame privilege optionsbullCan start as nologin then be granted loginbullCan cascade
bullAny role can be GRANTd to any other rolebullInherit is default acts like group privsbullNoinherit means user must run set role ala sudo
Permissionsbullpublic means all usersbullGRANT REVOKE to givetake away privs roles etcbullCONNECT privs on the database (public by default)bullschemas - CREATE USAGE
bullrecommend dropping public or revoke CREATEbullUse per-user or per-app schemas
bulltables - SELECTINSERTUPDATEDELETETRUNCATEbullview - same (incl update) execute as view ownerbullcolumns - SELECTINSERTUPDATEbullfunctions - SECURITY DEFINER are akin to setuid
Default permsbullGenerally secure-by-default
bullExcept functions- EXECUTE granted by defaultbullOwners have all rights on their objectsbullMembership in owning role == ownership
bullALTER DEFAULT PRIVILEGES - for rolesbullFOR ROLE IN SCHEMA GRANTbullCant be applied to just a schema
bullGRANT ON ALL IN SCHEMAbullFor tables views sequences functionsbullOne-time operation new tables will not have privs
TuningbullFor a dedicated serverbullshared_buffers
bullWill be dedicated to PG for cacheingbullUp to half of main memorybullTry 2G on larger servers more may not helpbullPre-93 need to bump sysctl paramsbullPost-93 you dontbullDefaults to 128MB
Tuning (continued)bullwork_mem
bullUsed for in-memory hashing sorts etcbullCan be increased inside a given connectionbullUsed many times over- not a hard limitbullPer connection so be carefulbullDefaults to 1MB (wayy too small)
bullmaintenance_work_membullUsed for building indexesbullMake it larger before building an indexbullDefaults to 16MB (thats a very small index)
Tuning (continued)bulleffective_cache_size
bullTells PG how much of the DB is in memorybullHalf of main memorybullNever allocated only for planning purposesbullDefaults to 128MB
bullautovacuumbullOn a high-rate server make it more aggressivebullIncrease max_workersbullDecrease autovacuum_vacuum_cost_delaybullDefaults are for lightly loaded systems
Config Bump-Upsbullmax_connections = 100
bullConsider using pg_bouncerbull connections == of CPUs is ideal
bullshared_buffers = couple gigbullProbably not more than 3-4G (Test)
bullmaintenance_work_mem = maybe a gigbullUsed for building indexes
bullmax_locks_per_transaction = 128bullMore if you have lots of objectsbull locks available is actually this max_conn
BackupsbullExtremely importantbullpg_basebackup w WAL recieve
bullBinary-based backupbullMUST have WAL files backed up alsobullNeeds to connect to replication DB
bullpg_dumpbullLogical text-based backupbullDoes not back up indexes must rebuildbullRequires lightweight locks on everything
bullTest restoring your data
Monitoringbullcheck_postgresplbullUseful with Nagios Icinga MRTG etcbullProvides metrics as well as monitoringbullAllows custom query for monitoringbullMinimum set of checksarchive_ready (if doing WAL archiving) --- Number of WAL ready filesautovac_freeze --- How close to Autovacuum Max Freezebackends (Metric) --- Number of Backends runningdbstats (Metrics) --- Lots of different statslistener (If using LISTENNOTIFY) --- Checks if anyone is LISTENinglocks (Metric) --- Number of locks heldpgbouncer options (if using pgbouncer) --- Various pgbouncer checkstxn_idle --- Transactions idle for X timetxn_time --- Transactions longer than X timetxn_wraparound --- How close to transaction wraparound
ExtensionsbullInstall -contrib packagebullUse PGXN - httppgxnorgbulltable pg_available_extensions name | default_version | installed_version | comment--------------------+-----------------+-------------------+---------------------------------------------------------------------- file_fdw | 10 | | foreign-data wrapper for flat file access dblink | 10 | | connect to other PostgreSQL databases from within a database plpgsql | 10 | 10 | PLpgSQL procedural language pg_trgm | 10 | | text similarity measurement and index searching based on trigrams adminpack | 10 | | administrative functions for PostgreSQL ip4r | 20 | | hstore | 11 | | data type for storing sets of (key value) pairs
bulladminpack allows superuser to change anythingbulldx lists installed extensions
Thank youStephen Frost
sfrostsnowmannetnet_snow
Loggingbullpostgresqlconf
bulllog_connections = onbulllog_disconnections = onbullline_prefix= m [p] q [l-1] d ur a bulllog_lock_waits = onbulllog_statement = ddlbulllog_min_duration_statement = 100bulllog_temp_files = 0bulllog_autovacuum_min_duration = 0
pg_hbaconfbullControls how users are authenticatedlocal DATABASE USER METHOD [OPTIONS]host DATABASE USER ADDRESS METHOD [OPTIONS]hostssl DATABASE USER ADDRESS METHOD [OPTIONS]hostnossl DATABASE USER ADDRESS METHOD [OPTIONS]
bullRead in order top-to-bottom first match is usedbullhostssl requires SSL connection no is not SSLbullSpecial DBs - all sameuser replicationbullSpecial Users - all + prefix for role membershipbullAddress can be IPv4 or IPv6 can include CIDR maskbullSpecial reject method
Authentication MethodsbullThe ones you should use bullpeer
bullSecure unix-socket-based authbullChecks the Unix username of the user
bullgss (Kerberos)bullIntegreates w MITHeimdal Kerberos and ADbullRecommended for Enterprise deployments
bullcert (SSL Certificate)bullClient-side certificate based authenticationbullUse pg_ident to map CNs to PG usernames
Authentication MethodsbullAcceptable but not idealbullmd5
bullStock usernamepasswordbullUse SSL if youre worried about security
bullpambullModules run as postgres userbullCant be used directly w pam_unixbullsaslauthd can make it work (pam_sasl saslauthd)
bullradiusbullUse SSL if youre worried about security
Auth Method Dontsbulltrust - Never use this- no auth donebullpassword - Password sent in cleartextbullsspi
bullWindows-specificbullUses KerberosGSSAPI underneath
bullidentbullInsecure dont trust it- use peer for local
bullldapbullAuths against an LDAP serverbullUse KerberosGSSAPI if you can
pg_identconfbullDefines mappings which are used in pg_hbamap-name auth-user pg-userkerbnames sfrostSNOWMANNET sfrostcertname stephenfrost sfrost
bullExternal-user to PG-user mappingsbullUnix user joe can be PG user bobbullRegexps can be used- but be carefulbullAlso works for Kerberos client certs etc
Debian configsbullExtra config files in DebianUbuntubullstartconf
bullControls start of this clusterbullCan be auto manual disabled
bullpg_ctlconfbullOptions to pass to pg_ctlbullGenerally dont need to modify it
bullenvironmentbullControlls environment PG starts inbullGenerally dont need to modify it
RedHat configsbullBasically just the initd scripts
Connectingbullsudo su - postgresbullpsqlbull to see backslash-commandsbullh to get help on SQL queriescommandsbullExit with q or ctrl-dbullpsql -h localhost
Looking aroundbulltable pg_stat_activity - aka wbulll - list databases Name | Owner | Encoding | Collate | Ctype | Access privileges-----------+----------+----------+-------------+-------------+----------------------- postgres | postgres | UTF8 | en_USUTF-8 | en_USUTF-8 | template0 | postgres | UTF8 | en_USUTF-8 | en_USUTF-8 | =cpostgres + template1 | postgres | UTF8 | en_USUTF-8 | en_USUTF-8 | =cpostgres +
bulldn - list schemas Name | Owner--------+---------- public | postgres
bulldb - list tablespaces Name | Owner | Location------------+----------+---------- pg_default | postgres | pg_global | postgres |
User setupsbullcreateuser CREATE USERbullpassword to set passwordsbullPrivileges
bullSuperuser- Do not give this outbullCreateRole- Creation and modification of rolesbullCreateDatabase- Allows database creationbullLogin- Allows user to connect to DBbullReplication- Only for replicationsystem userbullAdmin- Allows changing role membershipsbullInherit- Automatically get privileges
RolesbullUsers are really rolesbullGroups are implemented with rolesbullCREATE ROLE (or just createuser --nologin)
bullSame privilege optionsbullCan start as nologin then be granted loginbullCan cascade
bullAny role can be GRANTd to any other rolebullInherit is default acts like group privsbullNoinherit means user must run set role ala sudo
Permissionsbullpublic means all usersbullGRANT REVOKE to givetake away privs roles etcbullCONNECT privs on the database (public by default)bullschemas - CREATE USAGE
bullrecommend dropping public or revoke CREATEbullUse per-user or per-app schemas
bulltables - SELECTINSERTUPDATEDELETETRUNCATEbullview - same (incl update) execute as view ownerbullcolumns - SELECTINSERTUPDATEbullfunctions - SECURITY DEFINER are akin to setuid
Default permsbullGenerally secure-by-default
bullExcept functions- EXECUTE granted by defaultbullOwners have all rights on their objectsbullMembership in owning role == ownership
bullALTER DEFAULT PRIVILEGES - for rolesbullFOR ROLE IN SCHEMA GRANTbullCant be applied to just a schema
bullGRANT ON ALL IN SCHEMAbullFor tables views sequences functionsbullOne-time operation new tables will not have privs
TuningbullFor a dedicated serverbullshared_buffers
bullWill be dedicated to PG for cacheingbullUp to half of main memorybullTry 2G on larger servers more may not helpbullPre-93 need to bump sysctl paramsbullPost-93 you dontbullDefaults to 128MB
Tuning (continued)bullwork_mem
bullUsed for in-memory hashing sorts etcbullCan be increased inside a given connectionbullUsed many times over- not a hard limitbullPer connection so be carefulbullDefaults to 1MB (wayy too small)
bullmaintenance_work_membullUsed for building indexesbullMake it larger before building an indexbullDefaults to 16MB (thats a very small index)
Tuning (continued)bulleffective_cache_size
bullTells PG how much of the DB is in memorybullHalf of main memorybullNever allocated only for planning purposesbullDefaults to 128MB
bullautovacuumbullOn a high-rate server make it more aggressivebullIncrease max_workersbullDecrease autovacuum_vacuum_cost_delaybullDefaults are for lightly loaded systems
Config Bump-Upsbullmax_connections = 100
bullConsider using pg_bouncerbull connections == of CPUs is ideal
bullshared_buffers = couple gigbullProbably not more than 3-4G (Test)
bullmaintenance_work_mem = maybe a gigbullUsed for building indexes
bullmax_locks_per_transaction = 128bullMore if you have lots of objectsbull locks available is actually this max_conn
BackupsbullExtremely importantbullpg_basebackup w WAL recieve
bullBinary-based backupbullMUST have WAL files backed up alsobullNeeds to connect to replication DB
bullpg_dumpbullLogical text-based backupbullDoes not back up indexes must rebuildbullRequires lightweight locks on everything
bullTest restoring your data
Monitoringbullcheck_postgresplbullUseful with Nagios Icinga MRTG etcbullProvides metrics as well as monitoringbullAllows custom query for monitoringbullMinimum set of checksarchive_ready (if doing WAL archiving) --- Number of WAL ready filesautovac_freeze --- How close to Autovacuum Max Freezebackends (Metric) --- Number of Backends runningdbstats (Metrics) --- Lots of different statslistener (If using LISTENNOTIFY) --- Checks if anyone is LISTENinglocks (Metric) --- Number of locks heldpgbouncer options (if using pgbouncer) --- Various pgbouncer checkstxn_idle --- Transactions idle for X timetxn_time --- Transactions longer than X timetxn_wraparound --- How close to transaction wraparound
ExtensionsbullInstall -contrib packagebullUse PGXN - httppgxnorgbulltable pg_available_extensions name | default_version | installed_version | comment--------------------+-----------------+-------------------+---------------------------------------------------------------------- file_fdw | 10 | | foreign-data wrapper for flat file access dblink | 10 | | connect to other PostgreSQL databases from within a database plpgsql | 10 | 10 | PLpgSQL procedural language pg_trgm | 10 | | text similarity measurement and index searching based on trigrams adminpack | 10 | | administrative functions for PostgreSQL ip4r | 20 | | hstore | 11 | | data type for storing sets of (key value) pairs
bulladminpack allows superuser to change anythingbulldx lists installed extensions
Thank youStephen Frost
sfrostsnowmannetnet_snow
pg_hbaconfbullControls how users are authenticatedlocal DATABASE USER METHOD [OPTIONS]host DATABASE USER ADDRESS METHOD [OPTIONS]hostssl DATABASE USER ADDRESS METHOD [OPTIONS]hostnossl DATABASE USER ADDRESS METHOD [OPTIONS]
bullRead in order top-to-bottom first match is usedbullhostssl requires SSL connection no is not SSLbullSpecial DBs - all sameuser replicationbullSpecial Users - all + prefix for role membershipbullAddress can be IPv4 or IPv6 can include CIDR maskbullSpecial reject method
Authentication MethodsbullThe ones you should use bullpeer
bullSecure unix-socket-based authbullChecks the Unix username of the user
bullgss (Kerberos)bullIntegreates w MITHeimdal Kerberos and ADbullRecommended for Enterprise deployments
bullcert (SSL Certificate)bullClient-side certificate based authenticationbullUse pg_ident to map CNs to PG usernames
Authentication MethodsbullAcceptable but not idealbullmd5
bullStock usernamepasswordbullUse SSL if youre worried about security
bullpambullModules run as postgres userbullCant be used directly w pam_unixbullsaslauthd can make it work (pam_sasl saslauthd)
bullradiusbullUse SSL if youre worried about security
Auth Method Dontsbulltrust - Never use this- no auth donebullpassword - Password sent in cleartextbullsspi
bullWindows-specificbullUses KerberosGSSAPI underneath
bullidentbullInsecure dont trust it- use peer for local
bullldapbullAuths against an LDAP serverbullUse KerberosGSSAPI if you can
pg_identconfbullDefines mappings which are used in pg_hbamap-name auth-user pg-userkerbnames sfrostSNOWMANNET sfrostcertname stephenfrost sfrost
bullExternal-user to PG-user mappingsbullUnix user joe can be PG user bobbullRegexps can be used- but be carefulbullAlso works for Kerberos client certs etc
Debian configsbullExtra config files in DebianUbuntubullstartconf
bullControls start of this clusterbullCan be auto manual disabled
bullpg_ctlconfbullOptions to pass to pg_ctlbullGenerally dont need to modify it
bullenvironmentbullControlls environment PG starts inbullGenerally dont need to modify it
RedHat configsbullBasically just the initd scripts
Connectingbullsudo su - postgresbullpsqlbull to see backslash-commandsbullh to get help on SQL queriescommandsbullExit with q or ctrl-dbullpsql -h localhost
Looking aroundbulltable pg_stat_activity - aka wbulll - list databases Name | Owner | Encoding | Collate | Ctype | Access privileges-----------+----------+----------+-------------+-------------+----------------------- postgres | postgres | UTF8 | en_USUTF-8 | en_USUTF-8 | template0 | postgres | UTF8 | en_USUTF-8 | en_USUTF-8 | =cpostgres + template1 | postgres | UTF8 | en_USUTF-8 | en_USUTF-8 | =cpostgres +
bulldn - list schemas Name | Owner--------+---------- public | postgres
bulldb - list tablespaces Name | Owner | Location------------+----------+---------- pg_default | postgres | pg_global | postgres |
User setupsbullcreateuser CREATE USERbullpassword to set passwordsbullPrivileges
bullSuperuser- Do not give this outbullCreateRole- Creation and modification of rolesbullCreateDatabase- Allows database creationbullLogin- Allows user to connect to DBbullReplication- Only for replicationsystem userbullAdmin- Allows changing role membershipsbullInherit- Automatically get privileges
RolesbullUsers are really rolesbullGroups are implemented with rolesbullCREATE ROLE (or just createuser --nologin)
bullSame privilege optionsbullCan start as nologin then be granted loginbullCan cascade
bullAny role can be GRANTd to any other rolebullInherit is default acts like group privsbullNoinherit means user must run set role ala sudo
Permissionsbullpublic means all usersbullGRANT REVOKE to givetake away privs roles etcbullCONNECT privs on the database (public by default)bullschemas - CREATE USAGE
bullrecommend dropping public or revoke CREATEbullUse per-user or per-app schemas
bulltables - SELECTINSERTUPDATEDELETETRUNCATEbullview - same (incl update) execute as view ownerbullcolumns - SELECTINSERTUPDATEbullfunctions - SECURITY DEFINER are akin to setuid
Default permsbullGenerally secure-by-default
bullExcept functions- EXECUTE granted by defaultbullOwners have all rights on their objectsbullMembership in owning role == ownership
bullALTER DEFAULT PRIVILEGES - for rolesbullFOR ROLE IN SCHEMA GRANTbullCant be applied to just a schema
bullGRANT ON ALL IN SCHEMAbullFor tables views sequences functionsbullOne-time operation new tables will not have privs
TuningbullFor a dedicated serverbullshared_buffers
bullWill be dedicated to PG for cacheingbullUp to half of main memorybullTry 2G on larger servers more may not helpbullPre-93 need to bump sysctl paramsbullPost-93 you dontbullDefaults to 128MB
Tuning (continued)bullwork_mem
bullUsed for in-memory hashing sorts etcbullCan be increased inside a given connectionbullUsed many times over- not a hard limitbullPer connection so be carefulbullDefaults to 1MB (wayy too small)
bullmaintenance_work_membullUsed for building indexesbullMake it larger before building an indexbullDefaults to 16MB (thats a very small index)
Tuning (continued)bulleffective_cache_size
bullTells PG how much of the DB is in memorybullHalf of main memorybullNever allocated only for planning purposesbullDefaults to 128MB
bullautovacuumbullOn a high-rate server make it more aggressivebullIncrease max_workersbullDecrease autovacuum_vacuum_cost_delaybullDefaults are for lightly loaded systems
Config Bump-Upsbullmax_connections = 100
bullConsider using pg_bouncerbull connections == of CPUs is ideal
bullshared_buffers = couple gigbullProbably not more than 3-4G (Test)
bullmaintenance_work_mem = maybe a gigbullUsed for building indexes
bullmax_locks_per_transaction = 128bullMore if you have lots of objectsbull locks available is actually this max_conn
BackupsbullExtremely importantbullpg_basebackup w WAL recieve
bullBinary-based backupbullMUST have WAL files backed up alsobullNeeds to connect to replication DB
bullpg_dumpbullLogical text-based backupbullDoes not back up indexes must rebuildbullRequires lightweight locks on everything
bullTest restoring your data
Monitoringbullcheck_postgresplbullUseful with Nagios Icinga MRTG etcbullProvides metrics as well as monitoringbullAllows custom query for monitoringbullMinimum set of checksarchive_ready (if doing WAL archiving) --- Number of WAL ready filesautovac_freeze --- How close to Autovacuum Max Freezebackends (Metric) --- Number of Backends runningdbstats (Metrics) --- Lots of different statslistener (If using LISTENNOTIFY) --- Checks if anyone is LISTENinglocks (Metric) --- Number of locks heldpgbouncer options (if using pgbouncer) --- Various pgbouncer checkstxn_idle --- Transactions idle for X timetxn_time --- Transactions longer than X timetxn_wraparound --- How close to transaction wraparound
ExtensionsbullInstall -contrib packagebullUse PGXN - httppgxnorgbulltable pg_available_extensions name | default_version | installed_version | comment--------------------+-----------------+-------------------+---------------------------------------------------------------------- file_fdw | 10 | | foreign-data wrapper for flat file access dblink | 10 | | connect to other PostgreSQL databases from within a database plpgsql | 10 | 10 | PLpgSQL procedural language pg_trgm | 10 | | text similarity measurement and index searching based on trigrams adminpack | 10 | | administrative functions for PostgreSQL ip4r | 20 | | hstore | 11 | | data type for storing sets of (key value) pairs
bulladminpack allows superuser to change anythingbulldx lists installed extensions
Thank youStephen Frost
sfrostsnowmannetnet_snow
Authentication MethodsbullThe ones you should use bullpeer
bullSecure unix-socket-based authbullChecks the Unix username of the user
bullgss (Kerberos)bullIntegreates w MITHeimdal Kerberos and ADbullRecommended for Enterprise deployments
bullcert (SSL Certificate)bullClient-side certificate based authenticationbullUse pg_ident to map CNs to PG usernames
Authentication MethodsbullAcceptable but not idealbullmd5
bullStock usernamepasswordbullUse SSL if youre worried about security
bullpambullModules run as postgres userbullCant be used directly w pam_unixbullsaslauthd can make it work (pam_sasl saslauthd)
bullradiusbullUse SSL if youre worried about security
Auth Method Dontsbulltrust - Never use this- no auth donebullpassword - Password sent in cleartextbullsspi
bullWindows-specificbullUses KerberosGSSAPI underneath
bullidentbullInsecure dont trust it- use peer for local
bullldapbullAuths against an LDAP serverbullUse KerberosGSSAPI if you can
pg_identconfbullDefines mappings which are used in pg_hbamap-name auth-user pg-userkerbnames sfrostSNOWMANNET sfrostcertname stephenfrost sfrost
bullExternal-user to PG-user mappingsbullUnix user joe can be PG user bobbullRegexps can be used- but be carefulbullAlso works for Kerberos client certs etc
Debian configsbullExtra config files in DebianUbuntubullstartconf
bullControls start of this clusterbullCan be auto manual disabled
bullpg_ctlconfbullOptions to pass to pg_ctlbullGenerally dont need to modify it
bullenvironmentbullControlls environment PG starts inbullGenerally dont need to modify it
RedHat configsbullBasically just the initd scripts
Connectingbullsudo su - postgresbullpsqlbull to see backslash-commandsbullh to get help on SQL queriescommandsbullExit with q or ctrl-dbullpsql -h localhost
Looking aroundbulltable pg_stat_activity - aka wbulll - list databases Name | Owner | Encoding | Collate | Ctype | Access privileges-----------+----------+----------+-------------+-------------+----------------------- postgres | postgres | UTF8 | en_USUTF-8 | en_USUTF-8 | template0 | postgres | UTF8 | en_USUTF-8 | en_USUTF-8 | =cpostgres + template1 | postgres | UTF8 | en_USUTF-8 | en_USUTF-8 | =cpostgres +
bulldn - list schemas Name | Owner--------+---------- public | postgres
bulldb - list tablespaces Name | Owner | Location------------+----------+---------- pg_default | postgres | pg_global | postgres |
User setupsbullcreateuser CREATE USERbullpassword to set passwordsbullPrivileges
bullSuperuser- Do not give this outbullCreateRole- Creation and modification of rolesbullCreateDatabase- Allows database creationbullLogin- Allows user to connect to DBbullReplication- Only for replicationsystem userbullAdmin- Allows changing role membershipsbullInherit- Automatically get privileges
RolesbullUsers are really rolesbullGroups are implemented with rolesbullCREATE ROLE (or just createuser --nologin)
bullSame privilege optionsbullCan start as nologin then be granted loginbullCan cascade
bullAny role can be GRANTd to any other rolebullInherit is default acts like group privsbullNoinherit means user must run set role ala sudo
Permissionsbullpublic means all usersbullGRANT REVOKE to givetake away privs roles etcbullCONNECT privs on the database (public by default)bullschemas - CREATE USAGE
bullrecommend dropping public or revoke CREATEbullUse per-user or per-app schemas
bulltables - SELECTINSERTUPDATEDELETETRUNCATEbullview - same (incl update) execute as view ownerbullcolumns - SELECTINSERTUPDATEbullfunctions - SECURITY DEFINER are akin to setuid
Default permsbullGenerally secure-by-default
bullExcept functions- EXECUTE granted by defaultbullOwners have all rights on their objectsbullMembership in owning role == ownership
bullALTER DEFAULT PRIVILEGES - for rolesbullFOR ROLE IN SCHEMA GRANTbullCant be applied to just a schema
bullGRANT ON ALL IN SCHEMAbullFor tables views sequences functionsbullOne-time operation new tables will not have privs
TuningbullFor a dedicated serverbullshared_buffers
bullWill be dedicated to PG for cacheingbullUp to half of main memorybullTry 2G on larger servers more may not helpbullPre-93 need to bump sysctl paramsbullPost-93 you dontbullDefaults to 128MB
Tuning (continued)bullwork_mem
bullUsed for in-memory hashing sorts etcbullCan be increased inside a given connectionbullUsed many times over- not a hard limitbullPer connection so be carefulbullDefaults to 1MB (wayy too small)
bullmaintenance_work_membullUsed for building indexesbullMake it larger before building an indexbullDefaults to 16MB (thats a very small index)
Tuning (continued)bulleffective_cache_size
bullTells PG how much of the DB is in memorybullHalf of main memorybullNever allocated only for planning purposesbullDefaults to 128MB
bullautovacuumbullOn a high-rate server make it more aggressivebullIncrease max_workersbullDecrease autovacuum_vacuum_cost_delaybullDefaults are for lightly loaded systems
Config Bump-Upsbullmax_connections = 100
bullConsider using pg_bouncerbull connections == of CPUs is ideal
bullshared_buffers = couple gigbullProbably not more than 3-4G (Test)
bullmaintenance_work_mem = maybe a gigbullUsed for building indexes
bullmax_locks_per_transaction = 128bullMore if you have lots of objectsbull locks available is actually this max_conn
BackupsbullExtremely importantbullpg_basebackup w WAL recieve
bullBinary-based backupbullMUST have WAL files backed up alsobullNeeds to connect to replication DB
bullpg_dumpbullLogical text-based backupbullDoes not back up indexes must rebuildbullRequires lightweight locks on everything
bullTest restoring your data
Monitoringbullcheck_postgresplbullUseful with Nagios Icinga MRTG etcbullProvides metrics as well as monitoringbullAllows custom query for monitoringbullMinimum set of checksarchive_ready (if doing WAL archiving) --- Number of WAL ready filesautovac_freeze --- How close to Autovacuum Max Freezebackends (Metric) --- Number of Backends runningdbstats (Metrics) --- Lots of different statslistener (If using LISTENNOTIFY) --- Checks if anyone is LISTENinglocks (Metric) --- Number of locks heldpgbouncer options (if using pgbouncer) --- Various pgbouncer checkstxn_idle --- Transactions idle for X timetxn_time --- Transactions longer than X timetxn_wraparound --- How close to transaction wraparound
ExtensionsbullInstall -contrib packagebullUse PGXN - httppgxnorgbulltable pg_available_extensions name | default_version | installed_version | comment--------------------+-----------------+-------------------+---------------------------------------------------------------------- file_fdw | 10 | | foreign-data wrapper for flat file access dblink | 10 | | connect to other PostgreSQL databases from within a database plpgsql | 10 | 10 | PLpgSQL procedural language pg_trgm | 10 | | text similarity measurement and index searching based on trigrams adminpack | 10 | | administrative functions for PostgreSQL ip4r | 20 | | hstore | 11 | | data type for storing sets of (key value) pairs
bulladminpack allows superuser to change anythingbulldx lists installed extensions
Thank youStephen Frost
sfrostsnowmannetnet_snow
Authentication MethodsbullAcceptable but not idealbullmd5
bullStock usernamepasswordbullUse SSL if youre worried about security
bullpambullModules run as postgres userbullCant be used directly w pam_unixbullsaslauthd can make it work (pam_sasl saslauthd)
bullradiusbullUse SSL if youre worried about security
Auth Method Dontsbulltrust - Never use this- no auth donebullpassword - Password sent in cleartextbullsspi
bullWindows-specificbullUses KerberosGSSAPI underneath
bullidentbullInsecure dont trust it- use peer for local
bullldapbullAuths against an LDAP serverbullUse KerberosGSSAPI if you can
pg_identconfbullDefines mappings which are used in pg_hbamap-name auth-user pg-userkerbnames sfrostSNOWMANNET sfrostcertname stephenfrost sfrost
bullExternal-user to PG-user mappingsbullUnix user joe can be PG user bobbullRegexps can be used- but be carefulbullAlso works for Kerberos client certs etc
Debian configsbullExtra config files in DebianUbuntubullstartconf
bullControls start of this clusterbullCan be auto manual disabled
bullpg_ctlconfbullOptions to pass to pg_ctlbullGenerally dont need to modify it
bullenvironmentbullControlls environment PG starts inbullGenerally dont need to modify it
RedHat configsbullBasically just the initd scripts
Connectingbullsudo su - postgresbullpsqlbull to see backslash-commandsbullh to get help on SQL queriescommandsbullExit with q or ctrl-dbullpsql -h localhost
Looking aroundbulltable pg_stat_activity - aka wbulll - list databases Name | Owner | Encoding | Collate | Ctype | Access privileges-----------+----------+----------+-------------+-------------+----------------------- postgres | postgres | UTF8 | en_USUTF-8 | en_USUTF-8 | template0 | postgres | UTF8 | en_USUTF-8 | en_USUTF-8 | =cpostgres + template1 | postgres | UTF8 | en_USUTF-8 | en_USUTF-8 | =cpostgres +
bulldn - list schemas Name | Owner--------+---------- public | postgres
bulldb - list tablespaces Name | Owner | Location------------+----------+---------- pg_default | postgres | pg_global | postgres |
User setupsbullcreateuser CREATE USERbullpassword to set passwordsbullPrivileges
bullSuperuser- Do not give this outbullCreateRole- Creation and modification of rolesbullCreateDatabase- Allows database creationbullLogin- Allows user to connect to DBbullReplication- Only for replicationsystem userbullAdmin- Allows changing role membershipsbullInherit- Automatically get privileges
RolesbullUsers are really rolesbullGroups are implemented with rolesbullCREATE ROLE (or just createuser --nologin)
bullSame privilege optionsbullCan start as nologin then be granted loginbullCan cascade
bullAny role can be GRANTd to any other rolebullInherit is default acts like group privsbullNoinherit means user must run set role ala sudo
Permissionsbullpublic means all usersbullGRANT REVOKE to givetake away privs roles etcbullCONNECT privs on the database (public by default)bullschemas - CREATE USAGE
bullrecommend dropping public or revoke CREATEbullUse per-user or per-app schemas
bulltables - SELECTINSERTUPDATEDELETETRUNCATEbullview - same (incl update) execute as view ownerbullcolumns - SELECTINSERTUPDATEbullfunctions - SECURITY DEFINER are akin to setuid
Default permsbullGenerally secure-by-default
bullExcept functions- EXECUTE granted by defaultbullOwners have all rights on their objectsbullMembership in owning role == ownership
bullALTER DEFAULT PRIVILEGES - for rolesbullFOR ROLE IN SCHEMA GRANTbullCant be applied to just a schema
bullGRANT ON ALL IN SCHEMAbullFor tables views sequences functionsbullOne-time operation new tables will not have privs
TuningbullFor a dedicated serverbullshared_buffers
bullWill be dedicated to PG for cacheingbullUp to half of main memorybullTry 2G on larger servers more may not helpbullPre-93 need to bump sysctl paramsbullPost-93 you dontbullDefaults to 128MB
Tuning (continued)bullwork_mem
bullUsed for in-memory hashing sorts etcbullCan be increased inside a given connectionbullUsed many times over- not a hard limitbullPer connection so be carefulbullDefaults to 1MB (wayy too small)
bullmaintenance_work_membullUsed for building indexesbullMake it larger before building an indexbullDefaults to 16MB (thats a very small index)
Tuning (continued)bulleffective_cache_size
bullTells PG how much of the DB is in memorybullHalf of main memorybullNever allocated only for planning purposesbullDefaults to 128MB
bullautovacuumbullOn a high-rate server make it more aggressivebullIncrease max_workersbullDecrease autovacuum_vacuum_cost_delaybullDefaults are for lightly loaded systems
Config Bump-Upsbullmax_connections = 100
bullConsider using pg_bouncerbull connections == of CPUs is ideal
bullshared_buffers = couple gigbullProbably not more than 3-4G (Test)
bullmaintenance_work_mem = maybe a gigbullUsed for building indexes
bullmax_locks_per_transaction = 128bullMore if you have lots of objectsbull locks available is actually this max_conn
BackupsbullExtremely importantbullpg_basebackup w WAL recieve
bullBinary-based backupbullMUST have WAL files backed up alsobullNeeds to connect to replication DB
bullpg_dumpbullLogical text-based backupbullDoes not back up indexes must rebuildbullRequires lightweight locks on everything
bullTest restoring your data
Monitoringbullcheck_postgresplbullUseful with Nagios Icinga MRTG etcbullProvides metrics as well as monitoringbullAllows custom query for monitoringbullMinimum set of checksarchive_ready (if doing WAL archiving) --- Number of WAL ready filesautovac_freeze --- How close to Autovacuum Max Freezebackends (Metric) --- Number of Backends runningdbstats (Metrics) --- Lots of different statslistener (If using LISTENNOTIFY) --- Checks if anyone is LISTENinglocks (Metric) --- Number of locks heldpgbouncer options (if using pgbouncer) --- Various pgbouncer checkstxn_idle --- Transactions idle for X timetxn_time --- Transactions longer than X timetxn_wraparound --- How close to transaction wraparound
ExtensionsbullInstall -contrib packagebullUse PGXN - httppgxnorgbulltable pg_available_extensions name | default_version | installed_version | comment--------------------+-----------------+-------------------+---------------------------------------------------------------------- file_fdw | 10 | | foreign-data wrapper for flat file access dblink | 10 | | connect to other PostgreSQL databases from within a database plpgsql | 10 | 10 | PLpgSQL procedural language pg_trgm | 10 | | text similarity measurement and index searching based on trigrams adminpack | 10 | | administrative functions for PostgreSQL ip4r | 20 | | hstore | 11 | | data type for storing sets of (key value) pairs
bulladminpack allows superuser to change anythingbulldx lists installed extensions
Thank youStephen Frost
sfrostsnowmannetnet_snow
Auth Method Dontsbulltrust - Never use this- no auth donebullpassword - Password sent in cleartextbullsspi
bullWindows-specificbullUses KerberosGSSAPI underneath
bullidentbullInsecure dont trust it- use peer for local
bullldapbullAuths against an LDAP serverbullUse KerberosGSSAPI if you can
pg_identconfbullDefines mappings which are used in pg_hbamap-name auth-user pg-userkerbnames sfrostSNOWMANNET sfrostcertname stephenfrost sfrost
bullExternal-user to PG-user mappingsbullUnix user joe can be PG user bobbullRegexps can be used- but be carefulbullAlso works for Kerberos client certs etc
Debian configsbullExtra config files in DebianUbuntubullstartconf
bullControls start of this clusterbullCan be auto manual disabled
bullpg_ctlconfbullOptions to pass to pg_ctlbullGenerally dont need to modify it
bullenvironmentbullControlls environment PG starts inbullGenerally dont need to modify it
RedHat configsbullBasically just the initd scripts
Connectingbullsudo su - postgresbullpsqlbull to see backslash-commandsbullh to get help on SQL queriescommandsbullExit with q or ctrl-dbullpsql -h localhost
Looking aroundbulltable pg_stat_activity - aka wbulll - list databases Name | Owner | Encoding | Collate | Ctype | Access privileges-----------+----------+----------+-------------+-------------+----------------------- postgres | postgres | UTF8 | en_USUTF-8 | en_USUTF-8 | template0 | postgres | UTF8 | en_USUTF-8 | en_USUTF-8 | =cpostgres + template1 | postgres | UTF8 | en_USUTF-8 | en_USUTF-8 | =cpostgres +
bulldn - list schemas Name | Owner--------+---------- public | postgres
bulldb - list tablespaces Name | Owner | Location------------+----------+---------- pg_default | postgres | pg_global | postgres |
User setupsbullcreateuser CREATE USERbullpassword to set passwordsbullPrivileges
bullSuperuser- Do not give this outbullCreateRole- Creation and modification of rolesbullCreateDatabase- Allows database creationbullLogin- Allows user to connect to DBbullReplication- Only for replicationsystem userbullAdmin- Allows changing role membershipsbullInherit- Automatically get privileges
RolesbullUsers are really rolesbullGroups are implemented with rolesbullCREATE ROLE (or just createuser --nologin)
bullSame privilege optionsbullCan start as nologin then be granted loginbullCan cascade
bullAny role can be GRANTd to any other rolebullInherit is default acts like group privsbullNoinherit means user must run set role ala sudo
Permissionsbullpublic means all usersbullGRANT REVOKE to givetake away privs roles etcbullCONNECT privs on the database (public by default)bullschemas - CREATE USAGE
bullrecommend dropping public or revoke CREATEbullUse per-user or per-app schemas
bulltables - SELECTINSERTUPDATEDELETETRUNCATEbullview - same (incl update) execute as view ownerbullcolumns - SELECTINSERTUPDATEbullfunctions - SECURITY DEFINER are akin to setuid
Default permsbullGenerally secure-by-default
bullExcept functions- EXECUTE granted by defaultbullOwners have all rights on their objectsbullMembership in owning role == ownership
bullALTER DEFAULT PRIVILEGES - for rolesbullFOR ROLE IN SCHEMA GRANTbullCant be applied to just a schema
bullGRANT ON ALL IN SCHEMAbullFor tables views sequences functionsbullOne-time operation new tables will not have privs
TuningbullFor a dedicated serverbullshared_buffers
bullWill be dedicated to PG for cacheingbullUp to half of main memorybullTry 2G on larger servers more may not helpbullPre-93 need to bump sysctl paramsbullPost-93 you dontbullDefaults to 128MB
Tuning (continued)bullwork_mem
bullUsed for in-memory hashing sorts etcbullCan be increased inside a given connectionbullUsed many times over- not a hard limitbullPer connection so be carefulbullDefaults to 1MB (wayy too small)
bullmaintenance_work_membullUsed for building indexesbullMake it larger before building an indexbullDefaults to 16MB (thats a very small index)
Tuning (continued)bulleffective_cache_size
bullTells PG how much of the DB is in memorybullHalf of main memorybullNever allocated only for planning purposesbullDefaults to 128MB
bullautovacuumbullOn a high-rate server make it more aggressivebullIncrease max_workersbullDecrease autovacuum_vacuum_cost_delaybullDefaults are for lightly loaded systems
Config Bump-Upsbullmax_connections = 100
bullConsider using pg_bouncerbull connections == of CPUs is ideal
bullshared_buffers = couple gigbullProbably not more than 3-4G (Test)
bullmaintenance_work_mem = maybe a gigbullUsed for building indexes
bullmax_locks_per_transaction = 128bullMore if you have lots of objectsbull locks available is actually this max_conn
BackupsbullExtremely importantbullpg_basebackup w WAL recieve
bullBinary-based backupbullMUST have WAL files backed up alsobullNeeds to connect to replication DB
bullpg_dumpbullLogical text-based backupbullDoes not back up indexes must rebuildbullRequires lightweight locks on everything
bullTest restoring your data
Monitoringbullcheck_postgresplbullUseful with Nagios Icinga MRTG etcbullProvides metrics as well as monitoringbullAllows custom query for monitoringbullMinimum set of checksarchive_ready (if doing WAL archiving) --- Number of WAL ready filesautovac_freeze --- How close to Autovacuum Max Freezebackends (Metric) --- Number of Backends runningdbstats (Metrics) --- Lots of different statslistener (If using LISTENNOTIFY) --- Checks if anyone is LISTENinglocks (Metric) --- Number of locks heldpgbouncer options (if using pgbouncer) --- Various pgbouncer checkstxn_idle --- Transactions idle for X timetxn_time --- Transactions longer than X timetxn_wraparound --- How close to transaction wraparound
ExtensionsbullInstall -contrib packagebullUse PGXN - httppgxnorgbulltable pg_available_extensions name | default_version | installed_version | comment--------------------+-----------------+-------------------+---------------------------------------------------------------------- file_fdw | 10 | | foreign-data wrapper for flat file access dblink | 10 | | connect to other PostgreSQL databases from within a database plpgsql | 10 | 10 | PLpgSQL procedural language pg_trgm | 10 | | text similarity measurement and index searching based on trigrams adminpack | 10 | | administrative functions for PostgreSQL ip4r | 20 | | hstore | 11 | | data type for storing sets of (key value) pairs
bulladminpack allows superuser to change anythingbulldx lists installed extensions
Thank youStephen Frost
sfrostsnowmannetnet_snow
pg_identconfbullDefines mappings which are used in pg_hbamap-name auth-user pg-userkerbnames sfrostSNOWMANNET sfrostcertname stephenfrost sfrost
bullExternal-user to PG-user mappingsbullUnix user joe can be PG user bobbullRegexps can be used- but be carefulbullAlso works for Kerberos client certs etc
Debian configsbullExtra config files in DebianUbuntubullstartconf
bullControls start of this clusterbullCan be auto manual disabled
bullpg_ctlconfbullOptions to pass to pg_ctlbullGenerally dont need to modify it
bullenvironmentbullControlls environment PG starts inbullGenerally dont need to modify it
RedHat configsbullBasically just the initd scripts
Connectingbullsudo su - postgresbullpsqlbull to see backslash-commandsbullh to get help on SQL queriescommandsbullExit with q or ctrl-dbullpsql -h localhost
Looking aroundbulltable pg_stat_activity - aka wbulll - list databases Name | Owner | Encoding | Collate | Ctype | Access privileges-----------+----------+----------+-------------+-------------+----------------------- postgres | postgres | UTF8 | en_USUTF-8 | en_USUTF-8 | template0 | postgres | UTF8 | en_USUTF-8 | en_USUTF-8 | =cpostgres + template1 | postgres | UTF8 | en_USUTF-8 | en_USUTF-8 | =cpostgres +
bulldn - list schemas Name | Owner--------+---------- public | postgres
bulldb - list tablespaces Name | Owner | Location------------+----------+---------- pg_default | postgres | pg_global | postgres |
User setupsbullcreateuser CREATE USERbullpassword to set passwordsbullPrivileges
bullSuperuser- Do not give this outbullCreateRole- Creation and modification of rolesbullCreateDatabase- Allows database creationbullLogin- Allows user to connect to DBbullReplication- Only for replicationsystem userbullAdmin- Allows changing role membershipsbullInherit- Automatically get privileges
RolesbullUsers are really rolesbullGroups are implemented with rolesbullCREATE ROLE (or just createuser --nologin)
bullSame privilege optionsbullCan start as nologin then be granted loginbullCan cascade
bullAny role can be GRANTd to any other rolebullInherit is default acts like group privsbullNoinherit means user must run set role ala sudo
Permissionsbullpublic means all usersbullGRANT REVOKE to givetake away privs roles etcbullCONNECT privs on the database (public by default)bullschemas - CREATE USAGE
bullrecommend dropping public or revoke CREATEbullUse per-user or per-app schemas
bulltables - SELECTINSERTUPDATEDELETETRUNCATEbullview - same (incl update) execute as view ownerbullcolumns - SELECTINSERTUPDATEbullfunctions - SECURITY DEFINER are akin to setuid
Default permsbullGenerally secure-by-default
bullExcept functions- EXECUTE granted by defaultbullOwners have all rights on their objectsbullMembership in owning role == ownership
bullALTER DEFAULT PRIVILEGES - for rolesbullFOR ROLE IN SCHEMA GRANTbullCant be applied to just a schema
bullGRANT ON ALL IN SCHEMAbullFor tables views sequences functionsbullOne-time operation new tables will not have privs
TuningbullFor a dedicated serverbullshared_buffers
bullWill be dedicated to PG for cacheingbullUp to half of main memorybullTry 2G on larger servers more may not helpbullPre-93 need to bump sysctl paramsbullPost-93 you dontbullDefaults to 128MB
Tuning (continued)bullwork_mem
bullUsed for in-memory hashing sorts etcbullCan be increased inside a given connectionbullUsed many times over- not a hard limitbullPer connection so be carefulbullDefaults to 1MB (wayy too small)
bullmaintenance_work_membullUsed for building indexesbullMake it larger before building an indexbullDefaults to 16MB (thats a very small index)
Tuning (continued)bulleffective_cache_size
bullTells PG how much of the DB is in memorybullHalf of main memorybullNever allocated only for planning purposesbullDefaults to 128MB
bullautovacuumbullOn a high-rate server make it more aggressivebullIncrease max_workersbullDecrease autovacuum_vacuum_cost_delaybullDefaults are for lightly loaded systems
Config Bump-Upsbullmax_connections = 100
bullConsider using pg_bouncerbull connections == of CPUs is ideal
bullshared_buffers = couple gigbullProbably not more than 3-4G (Test)
bullmaintenance_work_mem = maybe a gigbullUsed for building indexes
bullmax_locks_per_transaction = 128bullMore if you have lots of objectsbull locks available is actually this max_conn
BackupsbullExtremely importantbullpg_basebackup w WAL recieve
bullBinary-based backupbullMUST have WAL files backed up alsobullNeeds to connect to replication DB
bullpg_dumpbullLogical text-based backupbullDoes not back up indexes must rebuildbullRequires lightweight locks on everything
bullTest restoring your data
Monitoringbullcheck_postgresplbullUseful with Nagios Icinga MRTG etcbullProvides metrics as well as monitoringbullAllows custom query for monitoringbullMinimum set of checksarchive_ready (if doing WAL archiving) --- Number of WAL ready filesautovac_freeze --- How close to Autovacuum Max Freezebackends (Metric) --- Number of Backends runningdbstats (Metrics) --- Lots of different statslistener (If using LISTENNOTIFY) --- Checks if anyone is LISTENinglocks (Metric) --- Number of locks heldpgbouncer options (if using pgbouncer) --- Various pgbouncer checkstxn_idle --- Transactions idle for X timetxn_time --- Transactions longer than X timetxn_wraparound --- How close to transaction wraparound
ExtensionsbullInstall -contrib packagebullUse PGXN - httppgxnorgbulltable pg_available_extensions name | default_version | installed_version | comment--------------------+-----------------+-------------------+---------------------------------------------------------------------- file_fdw | 10 | | foreign-data wrapper for flat file access dblink | 10 | | connect to other PostgreSQL databases from within a database plpgsql | 10 | 10 | PLpgSQL procedural language pg_trgm | 10 | | text similarity measurement and index searching based on trigrams adminpack | 10 | | administrative functions for PostgreSQL ip4r | 20 | | hstore | 11 | | data type for storing sets of (key value) pairs
bulladminpack allows superuser to change anythingbulldx lists installed extensions
Thank youStephen Frost
sfrostsnowmannetnet_snow
Debian configsbullExtra config files in DebianUbuntubullstartconf
bullControls start of this clusterbullCan be auto manual disabled
bullpg_ctlconfbullOptions to pass to pg_ctlbullGenerally dont need to modify it
bullenvironmentbullControlls environment PG starts inbullGenerally dont need to modify it
RedHat configsbullBasically just the initd scripts
Connectingbullsudo su - postgresbullpsqlbull to see backslash-commandsbullh to get help on SQL queriescommandsbullExit with q or ctrl-dbullpsql -h localhost
Looking aroundbulltable pg_stat_activity - aka wbulll - list databases Name | Owner | Encoding | Collate | Ctype | Access privileges-----------+----------+----------+-------------+-------------+----------------------- postgres | postgres | UTF8 | en_USUTF-8 | en_USUTF-8 | template0 | postgres | UTF8 | en_USUTF-8 | en_USUTF-8 | =cpostgres + template1 | postgres | UTF8 | en_USUTF-8 | en_USUTF-8 | =cpostgres +
bulldn - list schemas Name | Owner--------+---------- public | postgres
bulldb - list tablespaces Name | Owner | Location------------+----------+---------- pg_default | postgres | pg_global | postgres |
User setupsbullcreateuser CREATE USERbullpassword to set passwordsbullPrivileges
bullSuperuser- Do not give this outbullCreateRole- Creation and modification of rolesbullCreateDatabase- Allows database creationbullLogin- Allows user to connect to DBbullReplication- Only for replicationsystem userbullAdmin- Allows changing role membershipsbullInherit- Automatically get privileges
RolesbullUsers are really rolesbullGroups are implemented with rolesbullCREATE ROLE (or just createuser --nologin)
bullSame privilege optionsbullCan start as nologin then be granted loginbullCan cascade
bullAny role can be GRANTd to any other rolebullInherit is default acts like group privsbullNoinherit means user must run set role ala sudo
Permissionsbullpublic means all usersbullGRANT REVOKE to givetake away privs roles etcbullCONNECT privs on the database (public by default)bullschemas - CREATE USAGE
bullrecommend dropping public or revoke CREATEbullUse per-user or per-app schemas
bulltables - SELECTINSERTUPDATEDELETETRUNCATEbullview - same (incl update) execute as view ownerbullcolumns - SELECTINSERTUPDATEbullfunctions - SECURITY DEFINER are akin to setuid
Default permsbullGenerally secure-by-default
bullExcept functions- EXECUTE granted by defaultbullOwners have all rights on their objectsbullMembership in owning role == ownership
bullALTER DEFAULT PRIVILEGES - for rolesbullFOR ROLE IN SCHEMA GRANTbullCant be applied to just a schema
bullGRANT ON ALL IN SCHEMAbullFor tables views sequences functionsbullOne-time operation new tables will not have privs
TuningbullFor a dedicated serverbullshared_buffers
bullWill be dedicated to PG for cacheingbullUp to half of main memorybullTry 2G on larger servers more may not helpbullPre-93 need to bump sysctl paramsbullPost-93 you dontbullDefaults to 128MB
Tuning (continued)bullwork_mem
bullUsed for in-memory hashing sorts etcbullCan be increased inside a given connectionbullUsed many times over- not a hard limitbullPer connection so be carefulbullDefaults to 1MB (wayy too small)
bullmaintenance_work_membullUsed for building indexesbullMake it larger before building an indexbullDefaults to 16MB (thats a very small index)
Tuning (continued)bulleffective_cache_size
bullTells PG how much of the DB is in memorybullHalf of main memorybullNever allocated only for planning purposesbullDefaults to 128MB
bullautovacuumbullOn a high-rate server make it more aggressivebullIncrease max_workersbullDecrease autovacuum_vacuum_cost_delaybullDefaults are for lightly loaded systems
Config Bump-Upsbullmax_connections = 100
bullConsider using pg_bouncerbull connections == of CPUs is ideal
bullshared_buffers = couple gigbullProbably not more than 3-4G (Test)
bullmaintenance_work_mem = maybe a gigbullUsed for building indexes
bullmax_locks_per_transaction = 128bullMore if you have lots of objectsbull locks available is actually this max_conn
BackupsbullExtremely importantbullpg_basebackup w WAL recieve
bullBinary-based backupbullMUST have WAL files backed up alsobullNeeds to connect to replication DB
bullpg_dumpbullLogical text-based backupbullDoes not back up indexes must rebuildbullRequires lightweight locks on everything
bullTest restoring your data
Monitoringbullcheck_postgresplbullUseful with Nagios Icinga MRTG etcbullProvides metrics as well as monitoringbullAllows custom query for monitoringbullMinimum set of checksarchive_ready (if doing WAL archiving) --- Number of WAL ready filesautovac_freeze --- How close to Autovacuum Max Freezebackends (Metric) --- Number of Backends runningdbstats (Metrics) --- Lots of different statslistener (If using LISTENNOTIFY) --- Checks if anyone is LISTENinglocks (Metric) --- Number of locks heldpgbouncer options (if using pgbouncer) --- Various pgbouncer checkstxn_idle --- Transactions idle for X timetxn_time --- Transactions longer than X timetxn_wraparound --- How close to transaction wraparound
ExtensionsbullInstall -contrib packagebullUse PGXN - httppgxnorgbulltable pg_available_extensions name | default_version | installed_version | comment--------------------+-----------------+-------------------+---------------------------------------------------------------------- file_fdw | 10 | | foreign-data wrapper for flat file access dblink | 10 | | connect to other PostgreSQL databases from within a database plpgsql | 10 | 10 | PLpgSQL procedural language pg_trgm | 10 | | text similarity measurement and index searching based on trigrams adminpack | 10 | | administrative functions for PostgreSQL ip4r | 20 | | hstore | 11 | | data type for storing sets of (key value) pairs
bulladminpack allows superuser to change anythingbulldx lists installed extensions
Thank youStephen Frost
sfrostsnowmannetnet_snow
RedHat configsbullBasically just the initd scripts
Connectingbullsudo su - postgresbullpsqlbull to see backslash-commandsbullh to get help on SQL queriescommandsbullExit with q or ctrl-dbullpsql -h localhost
Looking aroundbulltable pg_stat_activity - aka wbulll - list databases Name | Owner | Encoding | Collate | Ctype | Access privileges-----------+----------+----------+-------------+-------------+----------------------- postgres | postgres | UTF8 | en_USUTF-8 | en_USUTF-8 | template0 | postgres | UTF8 | en_USUTF-8 | en_USUTF-8 | =cpostgres + template1 | postgres | UTF8 | en_USUTF-8 | en_USUTF-8 | =cpostgres +
bulldn - list schemas Name | Owner--------+---------- public | postgres
bulldb - list tablespaces Name | Owner | Location------------+----------+---------- pg_default | postgres | pg_global | postgres |
User setupsbullcreateuser CREATE USERbullpassword to set passwordsbullPrivileges
bullSuperuser- Do not give this outbullCreateRole- Creation and modification of rolesbullCreateDatabase- Allows database creationbullLogin- Allows user to connect to DBbullReplication- Only for replicationsystem userbullAdmin- Allows changing role membershipsbullInherit- Automatically get privileges
RolesbullUsers are really rolesbullGroups are implemented with rolesbullCREATE ROLE (or just createuser --nologin)
bullSame privilege optionsbullCan start as nologin then be granted loginbullCan cascade
bullAny role can be GRANTd to any other rolebullInherit is default acts like group privsbullNoinherit means user must run set role ala sudo
Permissionsbullpublic means all usersbullGRANT REVOKE to givetake away privs roles etcbullCONNECT privs on the database (public by default)bullschemas - CREATE USAGE
bullrecommend dropping public or revoke CREATEbullUse per-user or per-app schemas
bulltables - SELECTINSERTUPDATEDELETETRUNCATEbullview - same (incl update) execute as view ownerbullcolumns - SELECTINSERTUPDATEbullfunctions - SECURITY DEFINER are akin to setuid
Default permsbullGenerally secure-by-default
bullExcept functions- EXECUTE granted by defaultbullOwners have all rights on their objectsbullMembership in owning role == ownership
bullALTER DEFAULT PRIVILEGES - for rolesbullFOR ROLE IN SCHEMA GRANTbullCant be applied to just a schema
bullGRANT ON ALL IN SCHEMAbullFor tables views sequences functionsbullOne-time operation new tables will not have privs
TuningbullFor a dedicated serverbullshared_buffers
bullWill be dedicated to PG for cacheingbullUp to half of main memorybullTry 2G on larger servers more may not helpbullPre-93 need to bump sysctl paramsbullPost-93 you dontbullDefaults to 128MB
Tuning (continued)bullwork_mem
bullUsed for in-memory hashing sorts etcbullCan be increased inside a given connectionbullUsed many times over- not a hard limitbullPer connection so be carefulbullDefaults to 1MB (wayy too small)
bullmaintenance_work_membullUsed for building indexesbullMake it larger before building an indexbullDefaults to 16MB (thats a very small index)
Tuning (continued)bulleffective_cache_size
bullTells PG how much of the DB is in memorybullHalf of main memorybullNever allocated only for planning purposesbullDefaults to 128MB
bullautovacuumbullOn a high-rate server make it more aggressivebullIncrease max_workersbullDecrease autovacuum_vacuum_cost_delaybullDefaults are for lightly loaded systems
Config Bump-Upsbullmax_connections = 100
bullConsider using pg_bouncerbull connections == of CPUs is ideal
bullshared_buffers = couple gigbullProbably not more than 3-4G (Test)
bullmaintenance_work_mem = maybe a gigbullUsed for building indexes
bullmax_locks_per_transaction = 128bullMore if you have lots of objectsbull locks available is actually this max_conn
BackupsbullExtremely importantbullpg_basebackup w WAL recieve
bullBinary-based backupbullMUST have WAL files backed up alsobullNeeds to connect to replication DB
bullpg_dumpbullLogical text-based backupbullDoes not back up indexes must rebuildbullRequires lightweight locks on everything
bullTest restoring your data
Monitoringbullcheck_postgresplbullUseful with Nagios Icinga MRTG etcbullProvides metrics as well as monitoringbullAllows custom query for monitoringbullMinimum set of checksarchive_ready (if doing WAL archiving) --- Number of WAL ready filesautovac_freeze --- How close to Autovacuum Max Freezebackends (Metric) --- Number of Backends runningdbstats (Metrics) --- Lots of different statslistener (If using LISTENNOTIFY) --- Checks if anyone is LISTENinglocks (Metric) --- Number of locks heldpgbouncer options (if using pgbouncer) --- Various pgbouncer checkstxn_idle --- Transactions idle for X timetxn_time --- Transactions longer than X timetxn_wraparound --- How close to transaction wraparound
ExtensionsbullInstall -contrib packagebullUse PGXN - httppgxnorgbulltable pg_available_extensions name | default_version | installed_version | comment--------------------+-----------------+-------------------+---------------------------------------------------------------------- file_fdw | 10 | | foreign-data wrapper for flat file access dblink | 10 | | connect to other PostgreSQL databases from within a database plpgsql | 10 | 10 | PLpgSQL procedural language pg_trgm | 10 | | text similarity measurement and index searching based on trigrams adminpack | 10 | | administrative functions for PostgreSQL ip4r | 20 | | hstore | 11 | | data type for storing sets of (key value) pairs
bulladminpack allows superuser to change anythingbulldx lists installed extensions
Thank youStephen Frost
sfrostsnowmannetnet_snow
Connectingbullsudo su - postgresbullpsqlbull to see backslash-commandsbullh to get help on SQL queriescommandsbullExit with q or ctrl-dbullpsql -h localhost
Looking aroundbulltable pg_stat_activity - aka wbulll - list databases Name | Owner | Encoding | Collate | Ctype | Access privileges-----------+----------+----------+-------------+-------------+----------------------- postgres | postgres | UTF8 | en_USUTF-8 | en_USUTF-8 | template0 | postgres | UTF8 | en_USUTF-8 | en_USUTF-8 | =cpostgres + template1 | postgres | UTF8 | en_USUTF-8 | en_USUTF-8 | =cpostgres +
bulldn - list schemas Name | Owner--------+---------- public | postgres
bulldb - list tablespaces Name | Owner | Location------------+----------+---------- pg_default | postgres | pg_global | postgres |
User setupsbullcreateuser CREATE USERbullpassword to set passwordsbullPrivileges
bullSuperuser- Do not give this outbullCreateRole- Creation and modification of rolesbullCreateDatabase- Allows database creationbullLogin- Allows user to connect to DBbullReplication- Only for replicationsystem userbullAdmin- Allows changing role membershipsbullInherit- Automatically get privileges
RolesbullUsers are really rolesbullGroups are implemented with rolesbullCREATE ROLE (or just createuser --nologin)
bullSame privilege optionsbullCan start as nologin then be granted loginbullCan cascade
bullAny role can be GRANTd to any other rolebullInherit is default acts like group privsbullNoinherit means user must run set role ala sudo
Permissionsbullpublic means all usersbullGRANT REVOKE to givetake away privs roles etcbullCONNECT privs on the database (public by default)bullschemas - CREATE USAGE
bullrecommend dropping public or revoke CREATEbullUse per-user or per-app schemas
bulltables - SELECTINSERTUPDATEDELETETRUNCATEbullview - same (incl update) execute as view ownerbullcolumns - SELECTINSERTUPDATEbullfunctions - SECURITY DEFINER are akin to setuid
Default permsbullGenerally secure-by-default
bullExcept functions- EXECUTE granted by defaultbullOwners have all rights on their objectsbullMembership in owning role == ownership
bullALTER DEFAULT PRIVILEGES - for rolesbullFOR ROLE IN SCHEMA GRANTbullCant be applied to just a schema
bullGRANT ON ALL IN SCHEMAbullFor tables views sequences functionsbullOne-time operation new tables will not have privs
TuningbullFor a dedicated serverbullshared_buffers
bullWill be dedicated to PG for cacheingbullUp to half of main memorybullTry 2G on larger servers more may not helpbullPre-93 need to bump sysctl paramsbullPost-93 you dontbullDefaults to 128MB
Tuning (continued)bullwork_mem
bullUsed for in-memory hashing sorts etcbullCan be increased inside a given connectionbullUsed many times over- not a hard limitbullPer connection so be carefulbullDefaults to 1MB (wayy too small)
bullmaintenance_work_membullUsed for building indexesbullMake it larger before building an indexbullDefaults to 16MB (thats a very small index)
Tuning (continued)bulleffective_cache_size
bullTells PG how much of the DB is in memorybullHalf of main memorybullNever allocated only for planning purposesbullDefaults to 128MB
bullautovacuumbullOn a high-rate server make it more aggressivebullIncrease max_workersbullDecrease autovacuum_vacuum_cost_delaybullDefaults are for lightly loaded systems
Config Bump-Upsbullmax_connections = 100
bullConsider using pg_bouncerbull connections == of CPUs is ideal
bullshared_buffers = couple gigbullProbably not more than 3-4G (Test)
bullmaintenance_work_mem = maybe a gigbullUsed for building indexes
bullmax_locks_per_transaction = 128bullMore if you have lots of objectsbull locks available is actually this max_conn
BackupsbullExtremely importantbullpg_basebackup w WAL recieve
bullBinary-based backupbullMUST have WAL files backed up alsobullNeeds to connect to replication DB
bullpg_dumpbullLogical text-based backupbullDoes not back up indexes must rebuildbullRequires lightweight locks on everything
bullTest restoring your data
Monitoringbullcheck_postgresplbullUseful with Nagios Icinga MRTG etcbullProvides metrics as well as monitoringbullAllows custom query for monitoringbullMinimum set of checksarchive_ready (if doing WAL archiving) --- Number of WAL ready filesautovac_freeze --- How close to Autovacuum Max Freezebackends (Metric) --- Number of Backends runningdbstats (Metrics) --- Lots of different statslistener (If using LISTENNOTIFY) --- Checks if anyone is LISTENinglocks (Metric) --- Number of locks heldpgbouncer options (if using pgbouncer) --- Various pgbouncer checkstxn_idle --- Transactions idle for X timetxn_time --- Transactions longer than X timetxn_wraparound --- How close to transaction wraparound
ExtensionsbullInstall -contrib packagebullUse PGXN - httppgxnorgbulltable pg_available_extensions name | default_version | installed_version | comment--------------------+-----------------+-------------------+---------------------------------------------------------------------- file_fdw | 10 | | foreign-data wrapper for flat file access dblink | 10 | | connect to other PostgreSQL databases from within a database plpgsql | 10 | 10 | PLpgSQL procedural language pg_trgm | 10 | | text similarity measurement and index searching based on trigrams adminpack | 10 | | administrative functions for PostgreSQL ip4r | 20 | | hstore | 11 | | data type for storing sets of (key value) pairs
bulladminpack allows superuser to change anythingbulldx lists installed extensions
Thank youStephen Frost
sfrostsnowmannetnet_snow
Looking aroundbulltable pg_stat_activity - aka wbulll - list databases Name | Owner | Encoding | Collate | Ctype | Access privileges-----------+----------+----------+-------------+-------------+----------------------- postgres | postgres | UTF8 | en_USUTF-8 | en_USUTF-8 | template0 | postgres | UTF8 | en_USUTF-8 | en_USUTF-8 | =cpostgres + template1 | postgres | UTF8 | en_USUTF-8 | en_USUTF-8 | =cpostgres +
bulldn - list schemas Name | Owner--------+---------- public | postgres
bulldb - list tablespaces Name | Owner | Location------------+----------+---------- pg_default | postgres | pg_global | postgres |
User setupsbullcreateuser CREATE USERbullpassword to set passwordsbullPrivileges
bullSuperuser- Do not give this outbullCreateRole- Creation and modification of rolesbullCreateDatabase- Allows database creationbullLogin- Allows user to connect to DBbullReplication- Only for replicationsystem userbullAdmin- Allows changing role membershipsbullInherit- Automatically get privileges
RolesbullUsers are really rolesbullGroups are implemented with rolesbullCREATE ROLE (or just createuser --nologin)
bullSame privilege optionsbullCan start as nologin then be granted loginbullCan cascade
bullAny role can be GRANTd to any other rolebullInherit is default acts like group privsbullNoinherit means user must run set role ala sudo
Permissionsbullpublic means all usersbullGRANT REVOKE to givetake away privs roles etcbullCONNECT privs on the database (public by default)bullschemas - CREATE USAGE
bullrecommend dropping public or revoke CREATEbullUse per-user or per-app schemas
bulltables - SELECTINSERTUPDATEDELETETRUNCATEbullview - same (incl update) execute as view ownerbullcolumns - SELECTINSERTUPDATEbullfunctions - SECURITY DEFINER are akin to setuid
Default permsbullGenerally secure-by-default
bullExcept functions- EXECUTE granted by defaultbullOwners have all rights on their objectsbullMembership in owning role == ownership
bullALTER DEFAULT PRIVILEGES - for rolesbullFOR ROLE IN SCHEMA GRANTbullCant be applied to just a schema
bullGRANT ON ALL IN SCHEMAbullFor tables views sequences functionsbullOne-time operation new tables will not have privs
TuningbullFor a dedicated serverbullshared_buffers
bullWill be dedicated to PG for cacheingbullUp to half of main memorybullTry 2G on larger servers more may not helpbullPre-93 need to bump sysctl paramsbullPost-93 you dontbullDefaults to 128MB
Tuning (continued)bullwork_mem
bullUsed for in-memory hashing sorts etcbullCan be increased inside a given connectionbullUsed many times over- not a hard limitbullPer connection so be carefulbullDefaults to 1MB (wayy too small)
bullmaintenance_work_membullUsed for building indexesbullMake it larger before building an indexbullDefaults to 16MB (thats a very small index)
Tuning (continued)bulleffective_cache_size
bullTells PG how much of the DB is in memorybullHalf of main memorybullNever allocated only for planning purposesbullDefaults to 128MB
bullautovacuumbullOn a high-rate server make it more aggressivebullIncrease max_workersbullDecrease autovacuum_vacuum_cost_delaybullDefaults are for lightly loaded systems
Config Bump-Upsbullmax_connections = 100
bullConsider using pg_bouncerbull connections == of CPUs is ideal
bullshared_buffers = couple gigbullProbably not more than 3-4G (Test)
bullmaintenance_work_mem = maybe a gigbullUsed for building indexes
bullmax_locks_per_transaction = 128bullMore if you have lots of objectsbull locks available is actually this max_conn
BackupsbullExtremely importantbullpg_basebackup w WAL recieve
bullBinary-based backupbullMUST have WAL files backed up alsobullNeeds to connect to replication DB
bullpg_dumpbullLogical text-based backupbullDoes not back up indexes must rebuildbullRequires lightweight locks on everything
bullTest restoring your data
Monitoringbullcheck_postgresplbullUseful with Nagios Icinga MRTG etcbullProvides metrics as well as monitoringbullAllows custom query for monitoringbullMinimum set of checksarchive_ready (if doing WAL archiving) --- Number of WAL ready filesautovac_freeze --- How close to Autovacuum Max Freezebackends (Metric) --- Number of Backends runningdbstats (Metrics) --- Lots of different statslistener (If using LISTENNOTIFY) --- Checks if anyone is LISTENinglocks (Metric) --- Number of locks heldpgbouncer options (if using pgbouncer) --- Various pgbouncer checkstxn_idle --- Transactions idle for X timetxn_time --- Transactions longer than X timetxn_wraparound --- How close to transaction wraparound
ExtensionsbullInstall -contrib packagebullUse PGXN - httppgxnorgbulltable pg_available_extensions name | default_version | installed_version | comment--------------------+-----------------+-------------------+---------------------------------------------------------------------- file_fdw | 10 | | foreign-data wrapper for flat file access dblink | 10 | | connect to other PostgreSQL databases from within a database plpgsql | 10 | 10 | PLpgSQL procedural language pg_trgm | 10 | | text similarity measurement and index searching based on trigrams adminpack | 10 | | administrative functions for PostgreSQL ip4r | 20 | | hstore | 11 | | data type for storing sets of (key value) pairs
bulladminpack allows superuser to change anythingbulldx lists installed extensions
Thank youStephen Frost
sfrostsnowmannetnet_snow
User setupsbullcreateuser CREATE USERbullpassword to set passwordsbullPrivileges
bullSuperuser- Do not give this outbullCreateRole- Creation and modification of rolesbullCreateDatabase- Allows database creationbullLogin- Allows user to connect to DBbullReplication- Only for replicationsystem userbullAdmin- Allows changing role membershipsbullInherit- Automatically get privileges
RolesbullUsers are really rolesbullGroups are implemented with rolesbullCREATE ROLE (or just createuser --nologin)
bullSame privilege optionsbullCan start as nologin then be granted loginbullCan cascade
bullAny role can be GRANTd to any other rolebullInherit is default acts like group privsbullNoinherit means user must run set role ala sudo
Permissionsbullpublic means all usersbullGRANT REVOKE to givetake away privs roles etcbullCONNECT privs on the database (public by default)bullschemas - CREATE USAGE
bullrecommend dropping public or revoke CREATEbullUse per-user or per-app schemas
bulltables - SELECTINSERTUPDATEDELETETRUNCATEbullview - same (incl update) execute as view ownerbullcolumns - SELECTINSERTUPDATEbullfunctions - SECURITY DEFINER are akin to setuid
Default permsbullGenerally secure-by-default
bullExcept functions- EXECUTE granted by defaultbullOwners have all rights on their objectsbullMembership in owning role == ownership
bullALTER DEFAULT PRIVILEGES - for rolesbullFOR ROLE IN SCHEMA GRANTbullCant be applied to just a schema
bullGRANT ON ALL IN SCHEMAbullFor tables views sequences functionsbullOne-time operation new tables will not have privs
TuningbullFor a dedicated serverbullshared_buffers
bullWill be dedicated to PG for cacheingbullUp to half of main memorybullTry 2G on larger servers more may not helpbullPre-93 need to bump sysctl paramsbullPost-93 you dontbullDefaults to 128MB
Tuning (continued)bullwork_mem
bullUsed for in-memory hashing sorts etcbullCan be increased inside a given connectionbullUsed many times over- not a hard limitbullPer connection so be carefulbullDefaults to 1MB (wayy too small)
bullmaintenance_work_membullUsed for building indexesbullMake it larger before building an indexbullDefaults to 16MB (thats a very small index)
Tuning (continued)bulleffective_cache_size
bullTells PG how much of the DB is in memorybullHalf of main memorybullNever allocated only for planning purposesbullDefaults to 128MB
bullautovacuumbullOn a high-rate server make it more aggressivebullIncrease max_workersbullDecrease autovacuum_vacuum_cost_delaybullDefaults are for lightly loaded systems
Config Bump-Upsbullmax_connections = 100
bullConsider using pg_bouncerbull connections == of CPUs is ideal
bullshared_buffers = couple gigbullProbably not more than 3-4G (Test)
bullmaintenance_work_mem = maybe a gigbullUsed for building indexes
bullmax_locks_per_transaction = 128bullMore if you have lots of objectsbull locks available is actually this max_conn
BackupsbullExtremely importantbullpg_basebackup w WAL recieve
bullBinary-based backupbullMUST have WAL files backed up alsobullNeeds to connect to replication DB
bullpg_dumpbullLogical text-based backupbullDoes not back up indexes must rebuildbullRequires lightweight locks on everything
bullTest restoring your data
Monitoringbullcheck_postgresplbullUseful with Nagios Icinga MRTG etcbullProvides metrics as well as monitoringbullAllows custom query for monitoringbullMinimum set of checksarchive_ready (if doing WAL archiving) --- Number of WAL ready filesautovac_freeze --- How close to Autovacuum Max Freezebackends (Metric) --- Number of Backends runningdbstats (Metrics) --- Lots of different statslistener (If using LISTENNOTIFY) --- Checks if anyone is LISTENinglocks (Metric) --- Number of locks heldpgbouncer options (if using pgbouncer) --- Various pgbouncer checkstxn_idle --- Transactions idle for X timetxn_time --- Transactions longer than X timetxn_wraparound --- How close to transaction wraparound
ExtensionsbullInstall -contrib packagebullUse PGXN - httppgxnorgbulltable pg_available_extensions name | default_version | installed_version | comment--------------------+-----------------+-------------------+---------------------------------------------------------------------- file_fdw | 10 | | foreign-data wrapper for flat file access dblink | 10 | | connect to other PostgreSQL databases from within a database plpgsql | 10 | 10 | PLpgSQL procedural language pg_trgm | 10 | | text similarity measurement and index searching based on trigrams adminpack | 10 | | administrative functions for PostgreSQL ip4r | 20 | | hstore | 11 | | data type for storing sets of (key value) pairs
bulladminpack allows superuser to change anythingbulldx lists installed extensions
Thank youStephen Frost
sfrostsnowmannetnet_snow
RolesbullUsers are really rolesbullGroups are implemented with rolesbullCREATE ROLE (or just createuser --nologin)
bullSame privilege optionsbullCan start as nologin then be granted loginbullCan cascade
bullAny role can be GRANTd to any other rolebullInherit is default acts like group privsbullNoinherit means user must run set role ala sudo
Permissionsbullpublic means all usersbullGRANT REVOKE to givetake away privs roles etcbullCONNECT privs on the database (public by default)bullschemas - CREATE USAGE
bullrecommend dropping public or revoke CREATEbullUse per-user or per-app schemas
bulltables - SELECTINSERTUPDATEDELETETRUNCATEbullview - same (incl update) execute as view ownerbullcolumns - SELECTINSERTUPDATEbullfunctions - SECURITY DEFINER are akin to setuid
Default permsbullGenerally secure-by-default
bullExcept functions- EXECUTE granted by defaultbullOwners have all rights on their objectsbullMembership in owning role == ownership
bullALTER DEFAULT PRIVILEGES - for rolesbullFOR ROLE IN SCHEMA GRANTbullCant be applied to just a schema
bullGRANT ON ALL IN SCHEMAbullFor tables views sequences functionsbullOne-time operation new tables will not have privs
TuningbullFor a dedicated serverbullshared_buffers
bullWill be dedicated to PG for cacheingbullUp to half of main memorybullTry 2G on larger servers more may not helpbullPre-93 need to bump sysctl paramsbullPost-93 you dontbullDefaults to 128MB
Tuning (continued)bullwork_mem
bullUsed for in-memory hashing sorts etcbullCan be increased inside a given connectionbullUsed many times over- not a hard limitbullPer connection so be carefulbullDefaults to 1MB (wayy too small)
bullmaintenance_work_membullUsed for building indexesbullMake it larger before building an indexbullDefaults to 16MB (thats a very small index)
Tuning (continued)bulleffective_cache_size
bullTells PG how much of the DB is in memorybullHalf of main memorybullNever allocated only for planning purposesbullDefaults to 128MB
bullautovacuumbullOn a high-rate server make it more aggressivebullIncrease max_workersbullDecrease autovacuum_vacuum_cost_delaybullDefaults are for lightly loaded systems
Config Bump-Upsbullmax_connections = 100
bullConsider using pg_bouncerbull connections == of CPUs is ideal
bullshared_buffers = couple gigbullProbably not more than 3-4G (Test)
bullmaintenance_work_mem = maybe a gigbullUsed for building indexes
bullmax_locks_per_transaction = 128bullMore if you have lots of objectsbull locks available is actually this max_conn
BackupsbullExtremely importantbullpg_basebackup w WAL recieve
bullBinary-based backupbullMUST have WAL files backed up alsobullNeeds to connect to replication DB
bullpg_dumpbullLogical text-based backupbullDoes not back up indexes must rebuildbullRequires lightweight locks on everything
bullTest restoring your data
Monitoringbullcheck_postgresplbullUseful with Nagios Icinga MRTG etcbullProvides metrics as well as monitoringbullAllows custom query for monitoringbullMinimum set of checksarchive_ready (if doing WAL archiving) --- Number of WAL ready filesautovac_freeze --- How close to Autovacuum Max Freezebackends (Metric) --- Number of Backends runningdbstats (Metrics) --- Lots of different statslistener (If using LISTENNOTIFY) --- Checks if anyone is LISTENinglocks (Metric) --- Number of locks heldpgbouncer options (if using pgbouncer) --- Various pgbouncer checkstxn_idle --- Transactions idle for X timetxn_time --- Transactions longer than X timetxn_wraparound --- How close to transaction wraparound
ExtensionsbullInstall -contrib packagebullUse PGXN - httppgxnorgbulltable pg_available_extensions name | default_version | installed_version | comment--------------------+-----------------+-------------------+---------------------------------------------------------------------- file_fdw | 10 | | foreign-data wrapper for flat file access dblink | 10 | | connect to other PostgreSQL databases from within a database plpgsql | 10 | 10 | PLpgSQL procedural language pg_trgm | 10 | | text similarity measurement and index searching based on trigrams adminpack | 10 | | administrative functions for PostgreSQL ip4r | 20 | | hstore | 11 | | data type for storing sets of (key value) pairs
bulladminpack allows superuser to change anythingbulldx lists installed extensions
Thank youStephen Frost
sfrostsnowmannetnet_snow
Permissionsbullpublic means all usersbullGRANT REVOKE to givetake away privs roles etcbullCONNECT privs on the database (public by default)bullschemas - CREATE USAGE
bullrecommend dropping public or revoke CREATEbullUse per-user or per-app schemas
bulltables - SELECTINSERTUPDATEDELETETRUNCATEbullview - same (incl update) execute as view ownerbullcolumns - SELECTINSERTUPDATEbullfunctions - SECURITY DEFINER are akin to setuid
Default permsbullGenerally secure-by-default
bullExcept functions- EXECUTE granted by defaultbullOwners have all rights on their objectsbullMembership in owning role == ownership
bullALTER DEFAULT PRIVILEGES - for rolesbullFOR ROLE IN SCHEMA GRANTbullCant be applied to just a schema
bullGRANT ON ALL IN SCHEMAbullFor tables views sequences functionsbullOne-time operation new tables will not have privs
TuningbullFor a dedicated serverbullshared_buffers
bullWill be dedicated to PG for cacheingbullUp to half of main memorybullTry 2G on larger servers more may not helpbullPre-93 need to bump sysctl paramsbullPost-93 you dontbullDefaults to 128MB
Tuning (continued)bullwork_mem
bullUsed for in-memory hashing sorts etcbullCan be increased inside a given connectionbullUsed many times over- not a hard limitbullPer connection so be carefulbullDefaults to 1MB (wayy too small)
bullmaintenance_work_membullUsed for building indexesbullMake it larger before building an indexbullDefaults to 16MB (thats a very small index)
Tuning (continued)bulleffective_cache_size
bullTells PG how much of the DB is in memorybullHalf of main memorybullNever allocated only for planning purposesbullDefaults to 128MB
bullautovacuumbullOn a high-rate server make it more aggressivebullIncrease max_workersbullDecrease autovacuum_vacuum_cost_delaybullDefaults are for lightly loaded systems
Config Bump-Upsbullmax_connections = 100
bullConsider using pg_bouncerbull connections == of CPUs is ideal
bullshared_buffers = couple gigbullProbably not more than 3-4G (Test)
bullmaintenance_work_mem = maybe a gigbullUsed for building indexes
bullmax_locks_per_transaction = 128bullMore if you have lots of objectsbull locks available is actually this max_conn
BackupsbullExtremely importantbullpg_basebackup w WAL recieve
bullBinary-based backupbullMUST have WAL files backed up alsobullNeeds to connect to replication DB
bullpg_dumpbullLogical text-based backupbullDoes not back up indexes must rebuildbullRequires lightweight locks on everything
bullTest restoring your data
Monitoringbullcheck_postgresplbullUseful with Nagios Icinga MRTG etcbullProvides metrics as well as monitoringbullAllows custom query for monitoringbullMinimum set of checksarchive_ready (if doing WAL archiving) --- Number of WAL ready filesautovac_freeze --- How close to Autovacuum Max Freezebackends (Metric) --- Number of Backends runningdbstats (Metrics) --- Lots of different statslistener (If using LISTENNOTIFY) --- Checks if anyone is LISTENinglocks (Metric) --- Number of locks heldpgbouncer options (if using pgbouncer) --- Various pgbouncer checkstxn_idle --- Transactions idle for X timetxn_time --- Transactions longer than X timetxn_wraparound --- How close to transaction wraparound
ExtensionsbullInstall -contrib packagebullUse PGXN - httppgxnorgbulltable pg_available_extensions name | default_version | installed_version | comment--------------------+-----------------+-------------------+---------------------------------------------------------------------- file_fdw | 10 | | foreign-data wrapper for flat file access dblink | 10 | | connect to other PostgreSQL databases from within a database plpgsql | 10 | 10 | PLpgSQL procedural language pg_trgm | 10 | | text similarity measurement and index searching based on trigrams adminpack | 10 | | administrative functions for PostgreSQL ip4r | 20 | | hstore | 11 | | data type for storing sets of (key value) pairs
bulladminpack allows superuser to change anythingbulldx lists installed extensions
Thank youStephen Frost
sfrostsnowmannetnet_snow
Default permsbullGenerally secure-by-default
bullExcept functions- EXECUTE granted by defaultbullOwners have all rights on their objectsbullMembership in owning role == ownership
bullALTER DEFAULT PRIVILEGES - for rolesbullFOR ROLE IN SCHEMA GRANTbullCant be applied to just a schema
bullGRANT ON ALL IN SCHEMAbullFor tables views sequences functionsbullOne-time operation new tables will not have privs
TuningbullFor a dedicated serverbullshared_buffers
bullWill be dedicated to PG for cacheingbullUp to half of main memorybullTry 2G on larger servers more may not helpbullPre-93 need to bump sysctl paramsbullPost-93 you dontbullDefaults to 128MB
Tuning (continued)bullwork_mem
bullUsed for in-memory hashing sorts etcbullCan be increased inside a given connectionbullUsed many times over- not a hard limitbullPer connection so be carefulbullDefaults to 1MB (wayy too small)
bullmaintenance_work_membullUsed for building indexesbullMake it larger before building an indexbullDefaults to 16MB (thats a very small index)
Tuning (continued)bulleffective_cache_size
bullTells PG how much of the DB is in memorybullHalf of main memorybullNever allocated only for planning purposesbullDefaults to 128MB
bullautovacuumbullOn a high-rate server make it more aggressivebullIncrease max_workersbullDecrease autovacuum_vacuum_cost_delaybullDefaults are for lightly loaded systems
Config Bump-Upsbullmax_connections = 100
bullConsider using pg_bouncerbull connections == of CPUs is ideal
bullshared_buffers = couple gigbullProbably not more than 3-4G (Test)
bullmaintenance_work_mem = maybe a gigbullUsed for building indexes
bullmax_locks_per_transaction = 128bullMore if you have lots of objectsbull locks available is actually this max_conn
BackupsbullExtremely importantbullpg_basebackup w WAL recieve
bullBinary-based backupbullMUST have WAL files backed up alsobullNeeds to connect to replication DB
bullpg_dumpbullLogical text-based backupbullDoes not back up indexes must rebuildbullRequires lightweight locks on everything
bullTest restoring your data
Monitoringbullcheck_postgresplbullUseful with Nagios Icinga MRTG etcbullProvides metrics as well as monitoringbullAllows custom query for monitoringbullMinimum set of checksarchive_ready (if doing WAL archiving) --- Number of WAL ready filesautovac_freeze --- How close to Autovacuum Max Freezebackends (Metric) --- Number of Backends runningdbstats (Metrics) --- Lots of different statslistener (If using LISTENNOTIFY) --- Checks if anyone is LISTENinglocks (Metric) --- Number of locks heldpgbouncer options (if using pgbouncer) --- Various pgbouncer checkstxn_idle --- Transactions idle for X timetxn_time --- Transactions longer than X timetxn_wraparound --- How close to transaction wraparound
ExtensionsbullInstall -contrib packagebullUse PGXN - httppgxnorgbulltable pg_available_extensions name | default_version | installed_version | comment--------------------+-----------------+-------------------+---------------------------------------------------------------------- file_fdw | 10 | | foreign-data wrapper for flat file access dblink | 10 | | connect to other PostgreSQL databases from within a database plpgsql | 10 | 10 | PLpgSQL procedural language pg_trgm | 10 | | text similarity measurement and index searching based on trigrams adminpack | 10 | | administrative functions for PostgreSQL ip4r | 20 | | hstore | 11 | | data type for storing sets of (key value) pairs
bulladminpack allows superuser to change anythingbulldx lists installed extensions
Thank youStephen Frost
sfrostsnowmannetnet_snow
TuningbullFor a dedicated serverbullshared_buffers
bullWill be dedicated to PG for cacheingbullUp to half of main memorybullTry 2G on larger servers more may not helpbullPre-93 need to bump sysctl paramsbullPost-93 you dontbullDefaults to 128MB
Tuning (continued)bullwork_mem
bullUsed for in-memory hashing sorts etcbullCan be increased inside a given connectionbullUsed many times over- not a hard limitbullPer connection so be carefulbullDefaults to 1MB (wayy too small)
bullmaintenance_work_membullUsed for building indexesbullMake it larger before building an indexbullDefaults to 16MB (thats a very small index)
Tuning (continued)bulleffective_cache_size
bullTells PG how much of the DB is in memorybullHalf of main memorybullNever allocated only for planning purposesbullDefaults to 128MB
bullautovacuumbullOn a high-rate server make it more aggressivebullIncrease max_workersbullDecrease autovacuum_vacuum_cost_delaybullDefaults are for lightly loaded systems
Config Bump-Upsbullmax_connections = 100
bullConsider using pg_bouncerbull connections == of CPUs is ideal
bullshared_buffers = couple gigbullProbably not more than 3-4G (Test)
bullmaintenance_work_mem = maybe a gigbullUsed for building indexes
bullmax_locks_per_transaction = 128bullMore if you have lots of objectsbull locks available is actually this max_conn
BackupsbullExtremely importantbullpg_basebackup w WAL recieve
bullBinary-based backupbullMUST have WAL files backed up alsobullNeeds to connect to replication DB
bullpg_dumpbullLogical text-based backupbullDoes not back up indexes must rebuildbullRequires lightweight locks on everything
bullTest restoring your data
Monitoringbullcheck_postgresplbullUseful with Nagios Icinga MRTG etcbullProvides metrics as well as monitoringbullAllows custom query for monitoringbullMinimum set of checksarchive_ready (if doing WAL archiving) --- Number of WAL ready filesautovac_freeze --- How close to Autovacuum Max Freezebackends (Metric) --- Number of Backends runningdbstats (Metrics) --- Lots of different statslistener (If using LISTENNOTIFY) --- Checks if anyone is LISTENinglocks (Metric) --- Number of locks heldpgbouncer options (if using pgbouncer) --- Various pgbouncer checkstxn_idle --- Transactions idle for X timetxn_time --- Transactions longer than X timetxn_wraparound --- How close to transaction wraparound
ExtensionsbullInstall -contrib packagebullUse PGXN - httppgxnorgbulltable pg_available_extensions name | default_version | installed_version | comment--------------------+-----------------+-------------------+---------------------------------------------------------------------- file_fdw | 10 | | foreign-data wrapper for flat file access dblink | 10 | | connect to other PostgreSQL databases from within a database plpgsql | 10 | 10 | PLpgSQL procedural language pg_trgm | 10 | | text similarity measurement and index searching based on trigrams adminpack | 10 | | administrative functions for PostgreSQL ip4r | 20 | | hstore | 11 | | data type for storing sets of (key value) pairs
bulladminpack allows superuser to change anythingbulldx lists installed extensions
Thank youStephen Frost
sfrostsnowmannetnet_snow
Tuning (continued)bullwork_mem
bullUsed for in-memory hashing sorts etcbullCan be increased inside a given connectionbullUsed many times over- not a hard limitbullPer connection so be carefulbullDefaults to 1MB (wayy too small)
bullmaintenance_work_membullUsed for building indexesbullMake it larger before building an indexbullDefaults to 16MB (thats a very small index)
Tuning (continued)bulleffective_cache_size
bullTells PG how much of the DB is in memorybullHalf of main memorybullNever allocated only for planning purposesbullDefaults to 128MB
bullautovacuumbullOn a high-rate server make it more aggressivebullIncrease max_workersbullDecrease autovacuum_vacuum_cost_delaybullDefaults are for lightly loaded systems
Config Bump-Upsbullmax_connections = 100
bullConsider using pg_bouncerbull connections == of CPUs is ideal
bullshared_buffers = couple gigbullProbably not more than 3-4G (Test)
bullmaintenance_work_mem = maybe a gigbullUsed for building indexes
bullmax_locks_per_transaction = 128bullMore if you have lots of objectsbull locks available is actually this max_conn
BackupsbullExtremely importantbullpg_basebackup w WAL recieve
bullBinary-based backupbullMUST have WAL files backed up alsobullNeeds to connect to replication DB
bullpg_dumpbullLogical text-based backupbullDoes not back up indexes must rebuildbullRequires lightweight locks on everything
bullTest restoring your data
Monitoringbullcheck_postgresplbullUseful with Nagios Icinga MRTG etcbullProvides metrics as well as monitoringbullAllows custom query for monitoringbullMinimum set of checksarchive_ready (if doing WAL archiving) --- Number of WAL ready filesautovac_freeze --- How close to Autovacuum Max Freezebackends (Metric) --- Number of Backends runningdbstats (Metrics) --- Lots of different statslistener (If using LISTENNOTIFY) --- Checks if anyone is LISTENinglocks (Metric) --- Number of locks heldpgbouncer options (if using pgbouncer) --- Various pgbouncer checkstxn_idle --- Transactions idle for X timetxn_time --- Transactions longer than X timetxn_wraparound --- How close to transaction wraparound
ExtensionsbullInstall -contrib packagebullUse PGXN - httppgxnorgbulltable pg_available_extensions name | default_version | installed_version | comment--------------------+-----------------+-------------------+---------------------------------------------------------------------- file_fdw | 10 | | foreign-data wrapper for flat file access dblink | 10 | | connect to other PostgreSQL databases from within a database plpgsql | 10 | 10 | PLpgSQL procedural language pg_trgm | 10 | | text similarity measurement and index searching based on trigrams adminpack | 10 | | administrative functions for PostgreSQL ip4r | 20 | | hstore | 11 | | data type for storing sets of (key value) pairs
bulladminpack allows superuser to change anythingbulldx lists installed extensions
Thank youStephen Frost
sfrostsnowmannetnet_snow
Tuning (continued)bulleffective_cache_size
bullTells PG how much of the DB is in memorybullHalf of main memorybullNever allocated only for planning purposesbullDefaults to 128MB
bullautovacuumbullOn a high-rate server make it more aggressivebullIncrease max_workersbullDecrease autovacuum_vacuum_cost_delaybullDefaults are for lightly loaded systems
Config Bump-Upsbullmax_connections = 100
bullConsider using pg_bouncerbull connections == of CPUs is ideal
bullshared_buffers = couple gigbullProbably not more than 3-4G (Test)
bullmaintenance_work_mem = maybe a gigbullUsed for building indexes
bullmax_locks_per_transaction = 128bullMore if you have lots of objectsbull locks available is actually this max_conn
BackupsbullExtremely importantbullpg_basebackup w WAL recieve
bullBinary-based backupbullMUST have WAL files backed up alsobullNeeds to connect to replication DB
bullpg_dumpbullLogical text-based backupbullDoes not back up indexes must rebuildbullRequires lightweight locks on everything
bullTest restoring your data
Monitoringbullcheck_postgresplbullUseful with Nagios Icinga MRTG etcbullProvides metrics as well as monitoringbullAllows custom query for monitoringbullMinimum set of checksarchive_ready (if doing WAL archiving) --- Number of WAL ready filesautovac_freeze --- How close to Autovacuum Max Freezebackends (Metric) --- Number of Backends runningdbstats (Metrics) --- Lots of different statslistener (If using LISTENNOTIFY) --- Checks if anyone is LISTENinglocks (Metric) --- Number of locks heldpgbouncer options (if using pgbouncer) --- Various pgbouncer checkstxn_idle --- Transactions idle for X timetxn_time --- Transactions longer than X timetxn_wraparound --- How close to transaction wraparound
ExtensionsbullInstall -contrib packagebullUse PGXN - httppgxnorgbulltable pg_available_extensions name | default_version | installed_version | comment--------------------+-----------------+-------------------+---------------------------------------------------------------------- file_fdw | 10 | | foreign-data wrapper for flat file access dblink | 10 | | connect to other PostgreSQL databases from within a database plpgsql | 10 | 10 | PLpgSQL procedural language pg_trgm | 10 | | text similarity measurement and index searching based on trigrams adminpack | 10 | | administrative functions for PostgreSQL ip4r | 20 | | hstore | 11 | | data type for storing sets of (key value) pairs
bulladminpack allows superuser to change anythingbulldx lists installed extensions
Thank youStephen Frost
sfrostsnowmannetnet_snow
Config Bump-Upsbullmax_connections = 100
bullConsider using pg_bouncerbull connections == of CPUs is ideal
bullshared_buffers = couple gigbullProbably not more than 3-4G (Test)
bullmaintenance_work_mem = maybe a gigbullUsed for building indexes
bullmax_locks_per_transaction = 128bullMore if you have lots of objectsbull locks available is actually this max_conn
BackupsbullExtremely importantbullpg_basebackup w WAL recieve
bullBinary-based backupbullMUST have WAL files backed up alsobullNeeds to connect to replication DB
bullpg_dumpbullLogical text-based backupbullDoes not back up indexes must rebuildbullRequires lightweight locks on everything
bullTest restoring your data
Monitoringbullcheck_postgresplbullUseful with Nagios Icinga MRTG etcbullProvides metrics as well as monitoringbullAllows custom query for monitoringbullMinimum set of checksarchive_ready (if doing WAL archiving) --- Number of WAL ready filesautovac_freeze --- How close to Autovacuum Max Freezebackends (Metric) --- Number of Backends runningdbstats (Metrics) --- Lots of different statslistener (If using LISTENNOTIFY) --- Checks if anyone is LISTENinglocks (Metric) --- Number of locks heldpgbouncer options (if using pgbouncer) --- Various pgbouncer checkstxn_idle --- Transactions idle for X timetxn_time --- Transactions longer than X timetxn_wraparound --- How close to transaction wraparound
ExtensionsbullInstall -contrib packagebullUse PGXN - httppgxnorgbulltable pg_available_extensions name | default_version | installed_version | comment--------------------+-----------------+-------------------+---------------------------------------------------------------------- file_fdw | 10 | | foreign-data wrapper for flat file access dblink | 10 | | connect to other PostgreSQL databases from within a database plpgsql | 10 | 10 | PLpgSQL procedural language pg_trgm | 10 | | text similarity measurement and index searching based on trigrams adminpack | 10 | | administrative functions for PostgreSQL ip4r | 20 | | hstore | 11 | | data type for storing sets of (key value) pairs
bulladminpack allows superuser to change anythingbulldx lists installed extensions
Thank youStephen Frost
sfrostsnowmannetnet_snow
BackupsbullExtremely importantbullpg_basebackup w WAL recieve
bullBinary-based backupbullMUST have WAL files backed up alsobullNeeds to connect to replication DB
bullpg_dumpbullLogical text-based backupbullDoes not back up indexes must rebuildbullRequires lightweight locks on everything
bullTest restoring your data
Monitoringbullcheck_postgresplbullUseful with Nagios Icinga MRTG etcbullProvides metrics as well as monitoringbullAllows custom query for monitoringbullMinimum set of checksarchive_ready (if doing WAL archiving) --- Number of WAL ready filesautovac_freeze --- How close to Autovacuum Max Freezebackends (Metric) --- Number of Backends runningdbstats (Metrics) --- Lots of different statslistener (If using LISTENNOTIFY) --- Checks if anyone is LISTENinglocks (Metric) --- Number of locks heldpgbouncer options (if using pgbouncer) --- Various pgbouncer checkstxn_idle --- Transactions idle for X timetxn_time --- Transactions longer than X timetxn_wraparound --- How close to transaction wraparound
ExtensionsbullInstall -contrib packagebullUse PGXN - httppgxnorgbulltable pg_available_extensions name | default_version | installed_version | comment--------------------+-----------------+-------------------+---------------------------------------------------------------------- file_fdw | 10 | | foreign-data wrapper for flat file access dblink | 10 | | connect to other PostgreSQL databases from within a database plpgsql | 10 | 10 | PLpgSQL procedural language pg_trgm | 10 | | text similarity measurement and index searching based on trigrams adminpack | 10 | | administrative functions for PostgreSQL ip4r | 20 | | hstore | 11 | | data type for storing sets of (key value) pairs
bulladminpack allows superuser to change anythingbulldx lists installed extensions
Thank youStephen Frost
sfrostsnowmannetnet_snow
Monitoringbullcheck_postgresplbullUseful with Nagios Icinga MRTG etcbullProvides metrics as well as monitoringbullAllows custom query for monitoringbullMinimum set of checksarchive_ready (if doing WAL archiving) --- Number of WAL ready filesautovac_freeze --- How close to Autovacuum Max Freezebackends (Metric) --- Number of Backends runningdbstats (Metrics) --- Lots of different statslistener (If using LISTENNOTIFY) --- Checks if anyone is LISTENinglocks (Metric) --- Number of locks heldpgbouncer options (if using pgbouncer) --- Various pgbouncer checkstxn_idle --- Transactions idle for X timetxn_time --- Transactions longer than X timetxn_wraparound --- How close to transaction wraparound
ExtensionsbullInstall -contrib packagebullUse PGXN - httppgxnorgbulltable pg_available_extensions name | default_version | installed_version | comment--------------------+-----------------+-------------------+---------------------------------------------------------------------- file_fdw | 10 | | foreign-data wrapper for flat file access dblink | 10 | | connect to other PostgreSQL databases from within a database plpgsql | 10 | 10 | PLpgSQL procedural language pg_trgm | 10 | | text similarity measurement and index searching based on trigrams adminpack | 10 | | administrative functions for PostgreSQL ip4r | 20 | | hstore | 11 | | data type for storing sets of (key value) pairs
bulladminpack allows superuser to change anythingbulldx lists installed extensions
Thank youStephen Frost
sfrostsnowmannetnet_snow
ExtensionsbullInstall -contrib packagebullUse PGXN - httppgxnorgbulltable pg_available_extensions name | default_version | installed_version | comment--------------------+-----------------+-------------------+---------------------------------------------------------------------- file_fdw | 10 | | foreign-data wrapper for flat file access dblink | 10 | | connect to other PostgreSQL databases from within a database plpgsql | 10 | 10 | PLpgSQL procedural language pg_trgm | 10 | | text similarity measurement and index searching based on trigrams adminpack | 10 | | administrative functions for PostgreSQL ip4r | 20 | | hstore | 11 | | data type for storing sets of (key value) pairs
bulladminpack allows superuser to change anythingbulldx lists installed extensions
Thank youStephen Frost
sfrostsnowmannetnet_snow
Thank youStephen Frost
sfrostsnowmannetnet_snow
