PostgreSQL Server Programming - Second Editionindex-of.co.uk/Database/SQL Database/PostgreSQL... · PostgreSQL Server Programming Second Edition Credits About the Authors About the

PostgreSQLServerProgrammingSecondEdition

TableofContents


Credits

AbouttheAuthors

AbouttheReviewers

www.PacktPub.com

Supportfiles,eBooks,discountoffers,andmore

Whysubscribe?

FreeaccessforPacktaccountholders

Preface

Whatthisbookcovers

Whatyouneedforthisbook

Whothisbookisfor

Conventions

Readerfeedback

Customersupport

Downloadingtheexamplecode

Errata

Piracy

Questions

1.WhatIsaPostgreSQLServer?

Whyprogramintheserver?

UsingPL/pgSQLforintegritychecks

Aboutthisbook’scodeexamples

Switchingtotheexpandeddisplay

Movingbeyondsimplefunctions

Datacomparisonsusingoperators

Managingrelateddatawithtriggers

Auditingchanges

Datacleaning

Customsortorders

Programmingbestpractices

KISS–keepitsimplestupid

DRY–don’trepeatyourself

YAGNI–youain’tgonnaneedit

SOA–service-orientedarchitecture

Typeextensibility

Caching

Wrappingup–whyprogramintheserver?

Performance

Easeofmaintenance

Improvedproductivity

Simplewaystotightensecurity

Summary

2.ServerProgrammingEnvironments

Costofacquisition

Availabilityofdevelopers

Licensing

Predictability

Community

Procedurallanguages

Third-partytools

Platformcompatibility

Applicationdesign

Databasesareconsideredharmful

Encapsulation

WhatdoesPostgreSQLoffer?

Datalocality

Morebasics

Transactions

Generalerrorreportinganderrorhandling

User-definedfunctions

Otherparameters

Morecontrol

Summary

3.YourFirstPL/pgSQLFunction

WhyPL/pgSQL?

ThestructureofaPL/pgSQLfunction

Accessingfunctionarguments

Conditionalexpressions

Loopswithcounters

Statementtermination

Loopingthroughqueryresults

PERFORMversusSELECT

LoopingThroughArrays

Returningarecord

Actingonthefunction’sresults

Summary

4.ReturningStructuredData

Setsandarrays

Returningsets

Returningasetofintegers

Usingasetreturningfunction

Functionsbasedonviews

OUTparametersandrecords

OUTparameters

Returningrecords

UsingRETURNSTABLE

Returningwithnopredefinedstructure

ReturningSETOFANY

Variadicargumentlists

AsummaryoftheRETURNSETOFvariants

Returningcursors

Iteratingovercursorsreturnedfromanotherfunction

Wrappingupoffunctionsreturningcursors

Otherwaystoworkwithstructureddata

Complexdatatypesforthemodernworld–XMLandJSON

XMLdatatypeandreturningdataasXMLfromfunctions

ReturningdataintheJSONformat

Summary

5.PL/pgSQLTriggerFunctions

Creatingthetriggerfunction

Creatingthetrigger

Workingonasimple“Hey,I’mcalled”trigger

Theaudittrigger

DisallowingDELETE

DisallowingTRUNCATE

ModifyingtheNEWrecord

Thetimestampingtrigger

Theimmutablefieldstrigger

Controllingwhenatriggeriscalled

Conditionaltriggers

Triggersonspecificfieldchanges

Visibility

Mostimportantly–usetriggerscautiously!

VariablespassedtothePL/pgSQLTRIGGERfunction

Summary

6.PostgreSQLEventTriggers

Usecasesforcreatingeventtriggers

Creatingeventtriggers

Creatinganaudittrail

Preventingschemachanges

Aroadmapofeventtriggers

Summary

7.DebuggingPL/pgSQL

ManualdebuggingwithRAISENOTICE

Throwingexceptions

Loggingtoafile

TheadvantagesofRAISENOTICE

ThedisadvantagesofRAISENOTICE

Visualdebugging

Installingthedebugger

Installingthedebuggerfromthesource

InstallingpgAdmin3

Usingthedebugger

Theadvantagesofthedebugger

Thedisadvantagesofthedebugger

Summary

8.UsingUnrestrictedLanguages

Areuntrustedlanguagesinferiortotrustedones?

Canyouuseuntrustedlanguagesforimportantfunctions?

Willuntrustedlanguagescorruptthedatabase?

Whyuntrusted?

WhyPL/Python?

QuickintroductiontoPL/Python

AminimalPL/Pythonfunction

Datatypeconversions

WritingsimplefunctionsinPL/Python

Asimplefunction

Functionsreturningarecord

Tablefunctions

Runningqueriesinthedatabase

Runningsimplequeries

Usingpreparedqueries

Cachingpreparedqueries

WritingtriggerfunctionsinPL/Python

Exploringtheinputsofatrigger

Alogtrigger

Constructingqueries

Handlingexceptions

AtomicityinPython

DebuggingPL/Python

Usingplpy.notice()totrackthefunction’sprogress

Usingassert

Redirectingsys.stdoutandsys.stderr

Thinkingoutofthe“SQLdatabaseserver”box

Generatingthumbnailswhensavingimages

Sendingane-mail

Listingdirectorycontents

Summary

9.WritingAdvancedFunctionsinC

ThesimplestCfunction–return(a+b)

add_func.c

Version0callconventions

Makefile

CREATEFUNCTIONadd(int,int)

add_func.sql.in

SummaryforwritingaCfunction

Addingfunctionalitytoadd(int,int)

SmarthandlingofNULLarguments

Workingwithanynumberofarguments

BasicguidelinesforwritingCcode

Memoryallocation

Usepalloc()andpfree()

Zero-fillthestructures

Includefiles

Publicsymbolnames

ErrorreportingfromCfunctions

“Error”statesthatarenoterrors

Whenaremessagessenttotheclient?

RunningqueriesandcallingPostgreSQLfunctions

AsampleCfunctionusingSPI

Visibilityofdatachanges

MoreinfoonSPI_*functions

Handlingrecordsasargumentsorreturnedvalues

Returningasingletupleofacomplextype

Extractingfieldsfromanargumenttuple

Constructingareturntuple

Interlude–whatisDatum?

Returningasetofrecords

Fastcapturingofdatabasechanges

Doingsomethingatcommit/rollback

Synchronizingbetweenbackends

WritingfunctionsinC++

AdditionalresourcesforC

Summary

10.ScalingYourDatabasewithPL/Proxy

Creatingasimplesingle-serverchat

Dealingwithsuccess–splittingtablesovermultipledatabases

Whatexpansionplansworkandwhen?

Movingtoabiggerserver

Master-slavereplication–movingreadstoslave

Multimasterreplication

Datapartitioningacrossmultipleservers

Splittingthedata

PL/Proxy–thepartitioninglanguage

InstallingPL/Proxy

ThePL/Proxylanguagesyntax

CONNECT,CLUSTER,andRUNON

SELECTandTARGET

SPLIT–distributingarrayelementsoverseveralpartitions

Thedistributionofdata

ConfiguringthePL/Proxyclusterusingfunctions

ConfiguringthePL/ProxyclusterusingSQL/MED

Movingdatafromthesingletothepartitioneddatabase

ConnectionPooling

Summary

11.PL/Perl–PerlProceduralLanguage

WhentousePL/Perl

InstallingPL/Perl

AsimplePL/Perlfunction

Passingandreturningnon-scalartypes

WritingPL/Perltriggers

UntrustedPerl

Summary

12.PL/Tcl–TclProceduralLanguage

InstallingPL/Tcl

AsimplePL/Tclfunction

NullcheckingwithStrictfunctions

Theparameterformat

Passingandreturningarrays

Passingcomposite-typearguments

Accessingdatabases

WritingPL/Tcltriggers

UntrustedTcl

Summary

13.PublishingYourCodeasPostgreSQLExtensions

Whentocreateanextension

Unpackagedextensions

Extensionversions

The.controlfile

Buildinganextension

Installinganextension

Viewingextensions

Publishingyourextension

IntroductiontoPostgreSQLExtensionNetwork

Signinguptopublishyourextension

Creatinganextensionprojecttheeasyway

Providingthemetadataabouttheextension

Writingyourextensioncode

Creatingthepackage

SubmittingthepackagetoPGXN

InstallinganextensionfromPGXN

Summary

14.PostgreSQLasanExtensibleRDBMS

Whatcan’tbeextended?

Creatinganewoperator

Overloadinganoperator

Optimizingoperators

COMMUTATOR

NEGATOR

Creatingindexaccessmethods

Creatinguser-definedaggregates

Usingforeigndatawrappers

Summary

Index


PostgreSQLServerProgrammingSecondEditionCopyright©2015PacktPublishing

Allrightsreserved.Nopartofthisbookmaybereproduced,storedinaretrievalsystem,ortransmittedinanyformorbyanymeans,withoutthepriorwrittenpermissionofthepublisher,exceptinthecaseofbriefquotationsembeddedincriticalarticlesorreviews.

Everyefforthasbeenmadeinthepreparationofthisbooktoensuretheaccuracyoftheinformationpresented.However,theinformationcontainedinthisbookissoldwithoutwarranty,eitherexpressorimplied.Neithertheauthors,norPacktPublishing,anditsdealersanddistributorswillbeheldliableforanydamagescausedorallegedtobecauseddirectlyorindirectlybythisbook.

PacktPublishinghasendeavoredtoprovidetrademarkinformationaboutallofthecompaniesandproductsmentionedinthisbookbytheappropriateuseofcapitals.However,PacktPublishingcannotguaranteetheaccuracyofthisinformation.

Firstpublished:June2013

Secondedition:February2015

Productionreference:1210215

PublishedbyPacktPublishingLtd.

LiveryPlace

35LiveryStreet

BirminghamB32PB,UK.

ISBN978-1-78398-058-1

www.packtpub.com

http://www.packtpub.com

CreditsAuthors

UsamaDar

HannuKrosing

JimMlodgenski

KirkRoybal

Reviewers

StephenFrost

RickvanHattem

VibhorKumar

JeffLawson

MarianoReingart

JulienTachoires

CommissioningEditor

UshaIyer

AcquisitionEditors

AntonyLowe

MeetaRajani

SamWood

ContentDevelopmentEditor

AdrianRaposo

TechnicalEditors

MrunmayeePatil

ChinmayPuranik

CopyEditors

DiptiKapadia

AartiSaldanha

ProjectCoordinator

KinjalBari

Proofreaders

MariaGould

LindaMorris

Indexer

MonicaAjmeraMehta

ProductionCoordinator

NiteshThakur

CoverWork

NiteshThakur

AbouttheAuthorsUsamaDarisaseasonedsoftwaredeveloperandarchitect.Duringhis14years’career,hehasworkedextensivelywithPostgreSQLandotherdatabasetechnologies.HeworkedonPostgreSQLinternalsextensivelywhilehewasworkingforEnterpriseDB.Currently,helivesinMunichwhereheworksforHuawei’sEuropeanResearchCenter.Hedesignsthenextgenerationofhigh-performancedatabasesystemsbasedonopensourcetechnologies,suchasPostgreSQL,whichareusedunderhighworkloadsandstrictperformancerequirements.

HannuKrosingwasaPostgreSQLuserbeforeitwasrewrittentouseSQLasitsmainquerylanguagein1995.Therefore,hehasboththehistoricperspectiveofitsdevelopment,aswellasalmost20yearsofexperienceinusingittosolvevariousreal-lifeproblems.

HewasthefirstdatabaseadministratoranddatabasearchitectatSkype,whereheinventedtheshardinglanguagePL/Proxythatallowsyoutoscaletheuserdatabaseinordertoworkwithbillionsofusers.

AfterheleftSkypeattheendof2006—aboutayearafteritwasboughtbyeBay—hehasbeenworkingasaPostgreSQLconsultantwith2ndQuadrant,thepremierPostgreSQLconsultancywithaglobalreachandlocalpresenceinmostpartsoftheworld.

HehascoauthoredPostgreSQL9AdministrationCookbook,PacktPublishing,togetherwithoneofthemainPostgreSQLdevelopers,SimonRiggs.

Iwanttosincerelythankmywife,Evelyn,forhersupportwhilewritingthisbook.

JimMlodgenskiistheCTOofOpenSCG,aprofessionalservicescompanyfocusedonleveragingopensourcetechnologiesforstrategicadvantage.HewasformerlytheCEOofStormDB,adatabasecloudcompanyfocusedonhorizontalscalability.PriortoStormDB,hehasheldhighlytechnicalrolesatCirrusTechnology,Inc.,EnterpriseDB,andFusionTechnologies.

JimisalsoaferventadvocateofPostgreSQL.HeisontheboardoftheUnitedStatesPostgreSQLAssociationaswellasapartoftheorganizingteamsoftheNewYorkPostgreSQLUserGroupandPhiladelphiaPostgreSQLUserGroup.

KirkRoybalhasbeenanactivememberofthePostgreSQLcommunitysince1998.HehashelpedorganizeusergroupsinHouston,Dallas,andBloomington,IL.Hehasmentoredmanyjuniordatabaseadministratorsandprovidedcross-trainingtoseniordatabaseengineers.HehasprovidedsolutionsusingPostgreSQLforreporting,businessintelligence,datawarehousing,applications,anddevelopmentsupport.

HesawthescopeofPostgreSQLwhenhisfirstsmall-scalebusinesscustomeraskedforawebapplication.Atthattime,competitivedatabaseproductswereeitherextremelyimmatureorcostprohibitive.

KirkhasstoodbyhischoiceofPostgreSQLformanyyearsnow.Hisexpertiseisfoundedonkeepingupwithfeaturesandcapabilitiesastheybecomeavailable.

Writingabookhasbeenauniqueexperienceforme.Manypeoplefantasizeaboutit,fewstartone,andevenfewergettopublication.Iamproudtobepartofateamthatactuallymadeittothebookshelf(whichitselfisadiminishingbreed).ThankstoSarahCullingtonfromPacktPublishingforgivingmeachancetoparticipateintheproject.IbelievethatthePostgreSQLcommunitywillbebetterservedbythisinformation,andIhopethattheyreceivethisasarewardforthetimethattheyhaveinvestedinmeovertheyears.

Abookonlyhasthevaluethatthereadersgiveit.ThankyoutothePostgreSQLcommunityforallthetechnical,personal,andprofessionaldevelopmenthelpyouhaveprovided.ThePostgreSQLcommunityisagreatbunchofpeople,andIhaveenjoyedthecompanyofmanyofthem.Ihopetocontributemoretothisprojectinthefuture,andIhopeyoufindmycontributionsasvaluableasIfindyours.

Thankyoutomyfamilyforgivingmeareasontosucceedandforlisteningtothegobbledygookandnoddingappreciatively.

Haveyoueverhadyourfamilyaskyouwhatyouweredoingandansweredthemwithafunction?Tryit.No,thenagain,don’ttryit.Theymayjusthaveyouinvoluntarilycheckedinsomewhere.

AbouttheReviewersStephenFrostisamajorcontributorandcommittertoPostgreSQL,whohasbeeninvolvedwithPostgreSQLsince2002,andhasdevelopedfeaturessuchastherolesystemandcolumn-levelprivileges.

HeisthechieftechnologyofficeratCrunchyDataSolutions,Inc.,thePostgreSQLcompanyforSecureEnterprises.HeisinvolvedintheadvancementofPostgreSQL’scapabilities,particularlyintheareaofsecurityinordertosupporttheneedsofgovernmentandfinancialinstitutionswhohavestrictsecurityandregulatoryrequirements.

RickvanHattemisanentrepreneurwithacomputersciencebackgroundandalong-timeopensourcedeveloperwithvastexperienceintheC,C++,Python,andJavalanguages.Additionally,hehasworkedwithmostlargedatabaseserverssuchasOracle,MSSQL,andMySQL,buthehasbeenfocusingonPostgreSQLsinceVersion7.4.

HeisoneofthefoundersoftheFashiolista.comsocialnetwork,anduntilrecently,hewastheCTO.Here,heusedPostgreSQLtoscalethefeedsformillionsofuserstoshowthatPostgreSQLcanholduptoNoSQLsolutions,givensometuningandadditionaltools.AfterFashiolista,heworkedasafreelanceconsultantforseveralcompanies,including2ndQuadrant.

HeiscurrentlythefounderofPGMon.com,amonitoringservicethatanalyzesyourdatabases,indexes,andqueriestokeepthemrunningatpeakperformance.Inadditiontoanalyzingyourdatabasesettings,thesystemactivelymonitorsyourqueriesandgivesyourecommendationstoenhanceperformance.

Heisalsothecreatorandmaintainerofalargenumberofopensourceprojects,suchaspg_query_analyser,pg_cascade_timestamp,QtQuery,Python-Statsd,andDjango-Statsd.

VibhorKumarisaprincipalsystemarchitectatEnterpriseDBwhospecializesinassistingFortune100companiestodeploy,manage,andoptimizePostgresdatabases.HejoinedEnterpriseDBin2008toworkwithPostgresafterseveralyearsofworkingwithOraclesystems.HehasworkedinteamleadershiprolesatIBMGlobalServicesandBMCSoftwareaswellasanOracledatabaseadministratoratCMCLtd.forseveralyears.HehasdevelopedexpertiseinOracle,DB2,andMongoDBandholdscertificationsinthem.HehasexperienceworkingwithMSSQLServer,MySQL,anddatawarehousing.Heholdsabachelor’sdegreeincomputersciencefromtheUniversityofLucknowandamaster’sdegreeincomputersciencefromtheArmyInstituteofManagement,Kolkata.HeisacertifiedPostgreSQLtrainerandholdsaprofessionalcertificationinPostgresPlusAdvancedServerfromEnterpriseDB.

JeffLawsonhasbeenafananduserofPostgreSQLsincethetimehediscovereditin2001.Overtheyears,hehasalsodevelopedanddeployedapplicationsforIBMDB2,Oracle,MySQL,MicrosoftSQLServer,Sybase,andothers,buthealwaysprefersPostgreSQLforitsbalanceoffeaturesandopenness.MuchofhisexperienceinvolvesdevelopingforInternet-facingwebsites/projectsthatrequirehighlyscalabledatabaseswithhighavailabilityorwithprovisionsfordisasterrecovery.

HecurrentlyworksasthedirectorofsoftwaredevelopmentforFlightAware,whichisanairplane-trackingwebsitethatusesPostgreSQLandotheropensourcesoftwaretostoreandanalyzethepositionsofthethousandsofflightsthatareoperatedworldwideeveryday.Hehasextensiveexperienceinsoftwarearchitecture,datasecurity,andnetworkprotocoldesignfromthesoftwareengineeringpositionshehasheldatUniva/UnitedDevices,Microsoft,NASA’sJetPropulsionLaboratory,andWolfeTech.Heisafounderofdistributed.net,whichpioneereddistributedcomputinginthe1990s,andhecontinuestoserveasthechiefofoperationsandasamemberoftheboardthere.HeearnedaBScdegreeincomputersciencefromHarveyMuddCollege.

Heisfondofcattle,holdsanFAAprivatepilotcertificate,andownsanairplanebasedinHouston,Texas.

MarianoReingartlivesinBuenosAires,Argentina,andisaspecialistinthesoftwaredevelopmentofapplicationsandlibraries(webservices,PDF,GUI,replication,andsoon)withmorethan10yearsofexperience.Currently,heisthePostgreSQLregionalcontactforArgentinaandaPythonSoftwareFoundationmember.

Heisamajorcontributortotheweb2pyPythonwebframework,andnowhe’sworkingonthewxWidgetsmultiplatformGUItoolkit(specificallyintheQtportandAndroidmobileareas).Also,hehascontributedtomorethanadozenopensourceprojects,includinganinterfaceforFreeElectronicInvoicewebservices(PyAfipWs)andPythonicreplicationforPostgreSQL(PyReplica).

Hehasabachelor’sdegreeincomputersystemsanalysis,andcurrently,he’samaster’scandidatefortheMScinfreesoftwaredegreeattheOpenUniversityofCatalonia.

Heworksonhisownfundedentrepreneurialventureformedbyanopengroupofindependentprofessionals,dedicatedtosoftwaredevelopment,training,andtechnicalsupport,focusingonopensourcetools(GNU/Linux,Python,PostgreSQL,andweb2py/wxPython).

HehasworkedforlocalPython-basedcompaniesinlargebusinessapplications(ERP,SCM,andCRM)andmissioncriticalsystems(electioncounting,electronicvoting,and911emergencyeventssupport).Hehascontributedtobookssuchasweb2pyEnterpriseWebFramework,ThirdEdition,andweb2pyApplicationDevelopmentCookbook,PacktPublishing,andseveralSpanishtranslationsofthePostgreSQLofficialdocumentation.

Hisfullresumeisavailableathttp://reingart.blogspot.com/p/resume.html.

JulienTachoiresisaPostgreSQLspecialist,whoworksasconsultantfortheFrenchPostgreSQLcompanyDalibo.Heisthemaindeveloperofpg_activity,atop-endsoftwarededicatedtofollowthePostgreSQLincomingtrafficinrealtime,whichiswritteninPython.

IwanttothankmyemployerDalibo;mywife,Camille;andmyson,Arthur.

http://reingart.blogspot.com/p/resume.html

www.PacktPub.com

Supportfiles,eBooks,discountoffers,andmoreForsupportfilesanddownloadsrelatedtoyourbook,pleasevisitwww.PacktPub.com.

DidyouknowthatPacktofferseBookversionsofeverybookpublished,withPDFandePubfilesavailable?YoucanupgradetotheeBookversionatwww.PacktPub.comandasaprintbookcustomer,youareentitledtoadiscountontheeBookcopy.Getintouchwithusat<[email protected]>formoredetails.

Atwww.PacktPub.com,youcanalsoreadacollectionoffreetechnicalarticles,signupforarangeoffreenewslettersandreceiveexclusivediscountsandoffersonPacktbooksandeBooks.

https://www2.packtpub.com/books/subscription/packtlib

DoyouneedinstantsolutionstoyourITquestions?PacktLibisPackt’sonlinedigitalbooklibrary.Here,youcansearch,access,andreadPackt’sentirelibraryofbooks.

http://www.PacktPub.com


mailto:[email protected]


https://www2.packtpub.com/books/subscription/packtlib

Whysubscribe?FullysearchableacrosseverybookpublishedbyPacktCopyandpaste,print,andbookmarkcontentOndemandandaccessibleviaawebbrowser

FreeaccessforPacktaccountholdersIfyouhaveanaccountwithPacktatwww.PacktPub.com,youcanusethistoaccessPacktLibtodayandview9entirelyfreebooks.Simplyuseyourlogincredentialsforimmediateaccess.


PrefaceThisfascinatingguidetoserverprogrammingwilltakeyourskillsofPostgreSQLtoawholenewlevel.Astep-by-stepapproachwithilluminatingexampleswilleducateyouaboutthefullrangeofpossibilities.YouwillunderstandtheextensionframeworkofPostgreSQLandleverageitinwaysyouhaven’teveninventedyet.Youwilllearnhowtowritefunctionsandcreateyourowndatatypes,allinyourfavoriteprogramminglanguage.Itisastep-by-steptutorial,withplentyoftipsandtrickstokick-startserverprogramming.

WhatthisbookcoversChapter1,WhatIsaPostgreSQLServer?,introducesyoutothePostgreSQLserverandwillsetthetonefortherestofthebook.ItintroducesyoutothewaysinwhichaPostgreSQLserverisextendible,andshowsyouthatitcanbetreatedasacompletesoftwaredevelopmentframeworkinsteadofjustadatabaseserver.

Chapter2,ServerProgrammingEnvironments,elaboratesthatPostgreSQLisbuilttohandleuserneeds,butmoreimportantly,itisbuiltnottochangeunderneathusersinthefuture.ItwilltouchupontheenvironmentsandwillhighlightsomeoftheimportantthingstobekeptinmindwhenprogrammingontheserverinPostgreSQL.

Chapter3,YourFirstPL/pgSQLFunction,buildsthefoundationsbydemonstratinghowtowritesimplePL/pgSQLfunctions.

Chapter4,ReturningStructuredData,buildsontheknowledgeofwritingPL/pgSQLfunctionsanddemonstrateshowtowritefunctionsthatreturnasetofvaluessuchasrows,arrays,andcursors.

Chapter5,PL/pgSQLTriggerFunctions,discusseshowtowritePL/pgSQLfunctionsthatareusedtowritetriggerlogic.ItalsodiscussesthevarioustypesoftriggersavailableinPostgreSQLandtheoptionsthatadatabasedeveloperhaswhenwritingsuchfunctions.

Chapter6,PostgreSQLEventTriggers,discussesPostgreSQL’seventtriggerfunctionality.EventtriggersarefiredwhenrunningaDDLoperationonatable.ThischapterdiscussesthevariouspossibilitiesandoptionsofcreatingeventtriggersandtheirlimitationsinPostgreSQL.

Chapter7,DebuggingPL/pgSQL,elaboratesonhowtodebugPL/pgSQL’sstoredproceduresandfunctionsinPostgreSQL.ThischapterexplainshowtoinstallthedebuggerpluginandusethepgAdmindebuggerconsole.

Chapter8,UsingUnrestrictedLanguages,explainsthedifferencesbetweenrestrictedandunrestrictedPostgreSQLlanguages.ThischapterusesPL/PythonasanexampleanddemonstratestheexamplesofbothrestrictedandunrestrictedfunctionsinPL/Python.

Chapter9,WritingAdvancedFunctionsinC,explainshowtoextendPostgreSQLbywritinguser-definedfunctions(UDFs)inC.

Chapter10,ScalingYourDatabasewithPL/Proxy,explainstheuseofaspecialprogramminglanguageinPostgreSQLcalledPL/Proxyandhowtouseitinordertopartitionandshardyourdatabase.

Chapter11,PL/Perl–PerlProceduralLanguage,discussesapopularPLlanguageinPostgreSQLcalledPL/Perl.ThischapterusessomesimpleexamplestodemonstratehowyoucanusePerltowritedatabasefunctions.

Chapter12,PL/Tcl–TclProceduralLanguage,discussesTclasalanguageofchoicewhenwritingdatabasefunctions.ItdiscussestheprosandconsofusingTclinthedatabase.

Chapter13,PublishingYourCodeasPostgreSQLExtensions,discusseshowtopackageanddistributethePostgreSQLextensions.Well-packagedextensionscanbeeasilydistributedandinstalledbyotherusers.ThischapteralsointroducesyoutothePostgreSQLExtensionNetwork(PGXN)andshowsyouhowtouseittogettheextensionspublishedbyotherdevelopers.

Chapter14,PostgreSQLasanExtensibleRDBMS,discussesmoreextensibilityoptionsinPostgreSQL,suchascreatingnewdatatypes,operators,andindexmethods.

WhatyouneedforthisbookInordertofollowthisbook,youneedthefollowingsoftware:

PostgreSQLDatabaseServer9.4Linux/UnixOperatingSystemPython2,Perl,andTcl

WhothisbookisforThisbookisformoderatetoadvancedlevelPostgreSQLdatabaseprofessionals.Togetabetterunderstandingofthisbook,youshouldhaveageneralexperienceinwritingSQL,abasicideaofquerytuning,andsomecodingexperienceinalanguageofyourchoice.

ConventionsInthisbook,youwillfindanumberoftextstylesthatdistinguishbetweendifferentkindsofinformation.Herearesomeexamplesofthesestylesandanexplanationoftheirmeaning.

Codewordsintext,databasetablenames,foldernames,filenames,fileextensions,pathnames,dummyURLs,userinput,andTwitterhandlesareshownasfollows:“Ifanyofthechecksfail,youshoulddoROLLBACKinsteadofCOMMIT.”

Ablockofcodeissetasfollows:

CREATETABLEaccounts(ownertext,balancenumeric,amountnumeric);

INSERTINTOaccountsVALUES('Bob',100);

INSERTINTOaccountsVALUES('Mary',200);

Whenwewishtodrawyourattentiontoaparticularpartofacodeblock,therelevantlinesoritemsaresetinbold:

CREATEORREPLACEFUNCTIONfibonacci_seq(numinteger)

RETURNSSETOFintegerAS$$

DECLARE

aint:=0;

bint:=1;

BEGIN

IF(num<=0)

THENRETURN;

ENDIF;

RETURNNEXTa;

LOOP

EXITWHENnum<=1;

RETURNNEXTb;

num=num-1;

SELECTb,a+bINTOa,b;

ENDLOOP;

END;

$$LANGUAGEplpgsql;

Anycommand-lineinputoroutputiswrittenasfollows:

$psql-c"SELECT1AStest"

Newtermsandimportantwordsareshowninbold.Wordsthatyouseeonthescreen,forexample,inmenusordialogboxes,appearinthetextlikethis:“Entersomevaluesintothecolumns,asseenintheprecedingscreenshot,andclickontheDebugbutton.”

NoteWarningsorimportantnotesappearinaboxlikethis.

TipTipsandtricksappearlikethis.

ReaderfeedbackFeedbackfromourreadersisalwayswelcome.Letusknowwhatyouthinkaboutthisbook—whatyoulikedordisliked.Readerfeedbackisimportantforusasithelpsusdeveloptitlesthatyouwillreallygetthemostoutof.

Tosendusgeneralfeedback,simplye-mail<[email protected]>,andmentionthebook’stitleinthesubjectofyourmessage.

Ifthereisatopicthatyouhaveexpertiseinandyouareinterestedineitherwritingorcontributingtoabook,seeourauthorguideatwww.packtpub.com/authors.


http://www.packtpub.com/authors

CustomersupportNowthatyouaretheproudownerofaPacktbook,wehaveanumberofthingstohelpyoutogetthemostfromyourpurchase.

DownloadingtheexamplecodeYoucandownloadtheexamplecodefilesfromyouraccountathttp://www.packtpub.comforallthePacktPublishingbooksyouhavepurchased.Ifyoupurchasedthisbookelsewhere,youcanvisithttp://www.packtpub.com/supportandregistertohavethefilese-maileddirectlytoyou.


http://www.packtpub.com/support

ErrataAlthoughwehavetakeneverycaretoensuretheaccuracyofourcontent,mistakesdohappen.Ifyoufindamistakeinoneofourbooks—maybeamistakeinthetextorthecode—wewouldbegratefulifyoucouldreportthistous.Bydoingso,youcansaveotherreadersfromfrustrationandhelpusimprovesubsequentversionsofthisbook.Ifyoufindanyerrata,pleasereportthembyvisitinghttp://www.packtpub.com/submit-errata,selectingyourbook,clickingontheErrataSubmissionFormlink,andenteringthedetailsofyourerrata.Onceyourerrataareverified,yoursubmissionwillbeacceptedandtheerratawillbeuploadedtoourwebsiteoraddedtoanylistofexistingerrataundertheErratasectionofthattitle.

Toviewthepreviouslysubmittederrata,gotohttps://www.packtpub.com/books/content/supportandenterthenameofthebookinthesearchfield.TherequiredinformationwillappearundertheErratasection.

http://www.packtpub.com/submit-errata

https://www.packtpub.com/books/content/support

PiracyPiracyofcopyrightedmaterialontheInternetisanongoingproblemacrossallmedia.AtPackt,wetaketheprotectionofourcopyrightandlicensesveryseriously.IfyoucomeacrossanyillegalcopiesofourworksinanyformontheInternet,pleaseprovideuswiththelocationaddressorwebsitenameimmediatelysothatwecanpursuearemedy.

Pleasecontactusat<[email protected]>withalinktothesuspectedpiratedmaterial.

Weappreciateyourhelpinprotectingourauthorsandourabilitytobringyouvaluablecontent.


QuestionsIfyouhaveaproblemwithanyaspectofthisbook,youcancontactusat<[email protected]>,andwewilldoourbesttoaddresstheproblem.


Chapter1.WhatIsaPostgreSQLServer?IfyouthinkthataPostgreSQLServerisjustastoragesystemandtheonlywaytocommunicatewithitisbyexecutingSQLstatements,youarelimitingyourselftremendously.Thatis,youareusingjustatinypartofthedatabase’sfeatures.

APostgreSQLServerisapowerfulframeworkthatcanbeusedforallkindsofdataprocessing,andevensomenon-dataservertasks.Itisaserverplatformthatallowsyoutoeasilymixandmatchfunctionsandlibrariesfromseveralpopularlanguages.

Considerthiscomplicated,multilanguagesequenceofwork:

CallastringparsingfunctioninPerlConvertthestringtoXSLTandprocesstheresultusingJavaScriptAskforasecurestampfromanexternaltimestampingservice,suchashttp://guardtime.com/,usingtheirSDKforCWriteaPythonfunctiontodigitallysigntheresult

Thismultilanguagesequenceofworkcanbeimplementedasaseriesofsimplefunctioncallsusingseveraloftheavailableserverprogramminglanguages.ThedeveloperwhoneedstoaccomplishallthisworkcanjustcallasinglePostgreSQLfunctionwithouttheneedtobeawareofhowthedataisbeingpassedbetweenlanguagesandlibraries:

SELECTconvert_to_xslt_and_sign(raw_data_string);

Inthisbook,wewilldiscussseveralfacetsofPostgreSQLServerprogramming.PostgreSQLhasallofthenativeserver-sideprogrammingfeaturesavailableinmostlargerdatabasesystemssuchastriggers,whichareautomatedactionsinvokedautomaticallyeachtimedataischanged.However,ithasuniquelydeepabilitiestooverridethebuilt-inbehaviordowntoverybasicoperators.ThisuniquePostgreSQLabilitycomesfromitscatalog-drivendesign,whichstoresinformationaboutdatatypes,functions,andaccessmethods.TheabilityofPostgreSQLtoloaduser-definedfunctionsviadynamicloadingmakesitrapidlychangeablewithouthavingtorecompilethedatabaseitself.Thereareseveralthingsyoucandowiththisflexibilityofcustomization.Someexamplesofthiscustomizationincludethefollowing:

Writinguser-definedfunctions(UDF)tocarryoutcomplexcomputationsAddingcomplicatedconstraintstomakesurethatthedataintheservermeetsguidelinesCreatingtriggersinmanylanguagestomakerelatedchangestoothertables,auditchanges,forbidtheactionfromtakingplaceifitdoesnotmeetcertaincriteria,preventchangestothedatabase,enforceandexecutebusinessrules,orreplicatedataDefiningnewdatatypesandoperatorsinthedatabaseUsingthegeographytypesdefinedinthePostGISpackageAddingyourownindexaccessmethodsforeithertheexistingornewdatatypes,makingsomequeriesmuchmoreefficient

Whatsortofthingscanyoudowiththesefeatures?Therearelimitlesspossibilities,such

http://guardtime.com/

astheoneslistedhere:

Writedataextractorfunctionstogetjusttheinterestingpartsfromstructureddata,suchasXMLorJSON,withoutneedingtoshipthewhole,possiblyhuge,documenttotheclientapplication.Processeventsasynchronously,suchassendingmailswithoutslowingdownthemainapplication.Youcancreateamailqueueforchangestouserinformation,populatedbyatrigger.Aseparatemail-sendingprocesscanconsumethisdatawheneveritisnotifiedbyanapplicationprocess.Implementanewdatatypetocustomhashthepasswords.Writefunctions,whichprovideinsideinformationabouttheserver,forexample,cachecontents,table-wiselockinformation,ortheSSLcertificateinformationofaclientconnectionforamonitoringdashboard.

Therestofthischapterispresentedasaseriesofdescriptionsofcommondatamanagementtasks,showinghowtheycanbesolvedinarobustandelegantwayviaserverprogramming.

NoteThesamplesinthischapterarealltestedtowork,buttheycomewithminimalcommentary.Theyareusedherejusttoshowyouvariousthingsthatserverprogrammingcanaccomplish.Thetechniquesthataredescribedwillbeexplainedthoroughlyinlaterchapters.

Whyprogramintheserver?Developersprogramtheircodeinanumberofdifferentlanguages,anditcanbedesignedtorunjustaboutanywhere.Whenwritinganapplication,somepeoplefollowthephilosophythatasmuchofthelogicaspossiblefortheapplicationshouldbepushedtotheclient.WeseethisintheexplosionofapplicationsleveragingJavaScriptinsidebrowsers.Othersliketopushthelogicintothemiddletier,withanapplicationserverhandlingthebusinessrules.Theseareallvalidwaystodesignanapplication,sowhywillyouwanttoprograminthedatabaseserver?

Let’sstartwithasimpleexample.Manyapplicationsincludealistofcustomerswhohaveabalanceintheiraccount.We’llusethissampleschemaanddata:

CREATETABLEaccounts(ownertext,balancenumeric,amountnumeric);

INSERTINTOaccountsVALUES('Bob',100);

INSERTINTOaccountsVALUES('Mary',200);

TipDownloadingtheexamplecode

YoucandownloadtheexamplecodefilesforallthePacktbooksyouhavepurchasedfromyouraccountathttp://www.packtpub.com.Ifyoupurchasedthisbookelsewhere,youcanvisithttp://www.packtpub.com/supportandregistertohavethefilese-maileddirectlytoyou.

Whenusingadatabase,themostcommonwaytointeractwithit,istouseSQLqueries.Ifyouwanttomove14dollarsfromBob’saccounttoMary’saccountwithsimpleSQL,youcandosousingthefollowing:

UPDATEaccountsSETbalance=balance-14.00WHEREowner='Bob';

UPDATEaccountsSETbalance=balance+14.00WHEREowner='Mary';

However,youalsohavetomakesurethatBobactuallyhasenoughmoney(orcredit)inhisaccount.Notethatifanythingfails,thennoneofthetransactionswillhappen.Inanapplicationprogram,thisishowtheprecedingcodesnippetwillbemodified:

BEGIN;

SELECTamountFROMaccountsWHEREowner='Bob'FORUPDATE;—nowinthe

applicationcheckthattheamountisactuallybigger—than14

UPDATEaccountsSETamount=amount-14.00WHEREowner='Bob';

UPDATEaccountsSETamount=amount+14.00WHEREowner='Mary';

COMMIT;

DidMaryactuallyhaveanaccount?Ifshedidnot,thelastUPDATEcommandwillsucceedbyupdatingzerorows.Ifanyofthechecksfail,youshoulddoROLLBACKinsteadofCOMMIT.Onceyouhavedoneallthisforalltheclientsthattransfermoney,anewrequirementwillinvariablyarrive.Perhaps,theminimumamountthatcanbetransferredisnow5.00.Youwillneedtorevisitthecodeinallyourclientsagain.

So,whatcanyoudotomakeallofthismoremanageable,secure,androbust?Thisiswhereserverprogramming,executingcodeonthedatabaseserveritself,canhelp.Youcan


http://www.packtpub.com/support

movethecomputations,checks,anddatamanipulationsentirelyintoaUDFontheserver.Thisnotonlyensuresthatyouhaveonlyonecopyofoperationlogictomanage,butalsomakesthingsfasterbynotrequiringseveralroundtripsbetweentheclientandtheserver.Ifrequired,youcanalsomakesurethatonlytheessentialinformationisgivenoutfromthedatabase.Forexample,thereisnobusinessformostclientapplicationstoknowhowmuchmoneyBobhasinhisaccount.Mostly,theyonlyneedtoknowwhetherthereisenoughmoneytomakethetransfer,ortobemorespecific,whetherthetransactionsucceeded.

UsingPL/pgSQLforintegritychecksPostgreSQLincludesitsownprogramminglanguagenamedPL/pgSQLthatisaimedtointegrateeasilywithSQLcommands.PLstandsforprocedurallanguage,andthisisjustoneofthemanylanguagesavailableforwritingservercode.pgSQListheshorthandforPostgreSQL.

UnlikebasicSQL,PL/pgSQLincludesproceduralelements,suchastheabilitytousetheif/then/elsestatementsandloops.YoucaneasilyexecuteSQLstatements,orevenloopovertheresultofaSQLstatementinthelanguage.

TheintegritychecksneededfortheapplicationcanbedoneinaPL/pgSQLfunctionthattakesthreearguments:namesofthepayerandtherecipientandtheamounttobepaid.Thissamplealsoreturnsthestatusofthepayment:

CREATEORREPLACEFUNCTIONtransfer(

i_payertext,

i_recipienttext,

i_amountnumeric(15,2))

RETURNStext

AS

$$

DECLARE

payer_balnumeric;

BEGIN

SELECTbalanceINTOpayer_bal

FROMaccounts

WHEREowner=i_payerFORUPDATE;

IFNOTFOUNDTHEN

RETURN'Payeraccountnotfound';

ENDIF;

IFpayer_bal<i_amountTHEN

RETURN'Notenoughfunds';

ENDIF;

UPDATEaccounts

SETbalance=balance+i_amount

WHEREowner=i_recipient;

IFNOTFOUNDTHEN

RETURN'Recipientdoesnotexist';

ENDIF;

UPDATEaccounts

SETbalance=balance-i_amount

WHEREowner=i_payer;

RETURN'OK';

END;

$$LANGUAGEplpgsql;

Hereareafewexamplesoftheusageofthisfunction,assumingthatyouhaven’texecutedthepreviouslyproposedUPDATEstatementsyet:

postgres=#SELECT*FROMaccounts;

owner|balance

-------+---------

Bob|100

Mary|200

(2rows)

postgres=#SELECTtransfer('Bob','Mary',14.00);

transfer

----------

OK

(1row)

postgres=#SELECT*FROMaccounts;

owner|balance

-------+---------

Mary|214.00

Bob|86.00

(2rows)

Yourapplicationwillneedtocheckthereturncodeanddecidehowtohandletheseerrors.Aslongasitiswrittentorejectanyunexpectedvalue,youcanextendthisfunctiontodomorechecking,suchastheminimumtransferrableamount,andyoucanbesureitwillbeprevented.Thefollowingthreeerrorscanbereturned:

postgres=#SELECT*FROMtransfer('Fred','Mary',14.00);

transfer

-------------------------

Payeraccountnotfound

(1row)

postgres=#SELECT*FROMtransfer('Bob','Fred',14.00);

transfer

--------------------------

Recipientdoesnotexist

(1row)

postgres=#SELECT*FROMtransfer('Bob','Mary',500.00);

transfer

------------------

Notenoughfunds

(1row)

Forthesecheckstoalwayswork,youwillneedtomakeallthetransferoperationsgothroughthefunction,ratherthanmanuallychangingthevalueswithSQLstatements.Onewaytoachievethis,isbyrevokingupdateprivilegesfromusersandfromauserwithhigherprivilegesthatdefinethetransferfunctionwithSECURITYDEFINER.Thiswillallowtherestricteduserstorunthefunctionasiftheyhavehigherprivilegessimilartothefunction’screator.

Aboutthisbook’scodeexamplesThesampleoutputshownherehasbeencreatedwiththepsqlutilityofPostgreSQL,usuallyrunningonaLinuxsystem.MostofthecodewillworkthesamewayifyouareusingaGUIutilitysuchaspgAdmin3toaccesstheserverinstead.Takeanexampleofthefollowinglineofcode:

postgres=#SELECT1;

Thepostgres=#partisthepromptshownbythepsqlcommand.

TheexamplesinthisbookhavebeentestedusingPostgreSQL9.3.TheywillprobablyworkonPostgreSQLVersion8.3andlater.Therehaven’tbeenmanymajorchangestohowserverprogramminghappensinthelastfewversionsofPostgreSQL.Thesyntaxhasbecomestricterovertimetoreducethepossibilityofmistakesintheserverprogrammingcode.Duetothenatureofthesechanges,mostcodefromnewerversionswillstillrunontheolderones,unlessitusesverynewfeatures.However,theoldercodecaneasilyfailtorunduetooneofthenewlyenforcedrestrictions.

SwitchingtotheexpandeddisplayWhenusingthepsqlutilitytoexecuteaquery,PostgreSQLnormallyoutputstheresultusingverticallyalignedcolumns:

$psql-c"SELECT1AStest"

test

------

1

(1row)

$psql

psql(9.3.2)

Type"help"forhelp.

postgres=#SELECT1AStest;

test

------

1

(1row)

Youcantellwhenyou’reseeingaregularoutputbecauseitwillendupshowingthenumberofrows.

Thistypeofoutputishardtofitintothetextofabooksuchasthis.It’seasiertoprinttheoutputfromwhattheprogramcallstheexpandeddisplay,whichbreakseachcolumnintoaseparateline.Youcanswitchtotheexpandeddisplayusingeitherthe-xcommand-lineswitchorbysending\xtothepsqlprogram.Here’sanexampleofusingeachofthese:

$psql-x-c"SELECT1AStest"

-[RECORD1]

test|1

$psql

psql(9.3.2)

Type"help"forhelp.

postgres=#\x

Expandeddisplayison.

postgres=#SELECT1AStest;

-[RECORD1]

test|1

Noticehowtheexpandedoutputdoesn’tshowtherowcountandnumberseachoutputrow.Tosavespace,notalloftheexamplesinthebookwillshowtheexpandedoutputbeingturnedon.Youcannormallytellwhichtypeyoucansee,bydifferencessuchaswhetheryou’reseeingrowsorRECORD.Theexpandedmodewillnormallybepreferredwhentheoutputofthequeryistoowidetofitintotheavailablewidthofthebook.Itisagoodideatosettheexpandedmodetoauto.Thiswillautomaticallyswitchtoexpandedmodefortableswithalotofcolumns.Youcanturnontheexpandedmodeusing\xauto:

postgres=#\xauto

Expandeddisplayisusedautomatically.

MovingbeyondsimplefunctionsServerprogrammingcanmeanalotofdifferentthings.Serverprogrammingisnotjustaboutwritingserverfunctions.Therearemanyotherthingsyoucandointheserver,whichcanbeconsideredasprogramming.

DatacomparisonsusingoperatorsFormorecomplextasks,youcandefineyourowntypes,operators,andcastsfromonetypetoanother,lettingyouactuallycompareapplesandoranges.

Asshowninthenextexample,youcandefinethetypefruit_qtyforfruit-with-quantityandthenteachPostgreSQLtocompareapplesandoranges,saytomakeoneorangetobeworth1.5apples,inordertoconvertapplestooranges:

postgres=#CREATETYPEFRUIT_QTYas(nametext,qtyint);

postgres=#SELECT'("APPLE",3)'::FRUIT_QTY;

fruit_qty

----------------

(APPLE,3)

(1row)

CREATEFUNCTIONfruit_qty_larger_than(left_fruitFRUIT_QTY,right_fruit

FRUIT_QTY)

RETURNSBOOL

AS$$

BEGIN

IF(left_fruit.name='APPLE'ANDright_fruit.name='ORANGE')

THEN

RETURNleft_fruit.qty>(1.5*right_fruit.qty);

ENDIF;

IF(left_fruit.name='ORANGE'ANDright_fruit.name='APPLE')

THEN

RETURN(1.5*left_fruit.qty)>right_fruit.qty;

ENDIF;

RETURNleft_fruit.qty>right_fruit.qty;

END;

$$

LANGUAGEplpgsql;

postgres=#SELECTfruit_qty_larger_than('("APPLE",

3)'::FRUIT_QTY,'("ORANGE",2)'::FRUIT_QTY);

fruit_qty_larger_than

-----------------------

f

(1row)

postgres=#SELECTfruit_qty_larger_than('("APPLE",

4)'::FRUIT_QTY,'("ORANGE",2)'::FRUIT_QTY);

fruit_qty_larger_than

-----------------------

t

(1row)

CREATEOPERATOR>(

leftarg=FRUIT_QTY,

rightarg=FRUIT_QTY,

procedure=fruit_qty_larger_than,

commutator=>

);

postgres=#SELECT'("ORANGE",2)'::FRUIT_QTY>'("APPLE",2)'::FRUIT_QTY;

?column?

----------

t

(1row)

postgres=#SELECT'("ORANGE",2)'::FRUIT_QTY>'("APPLE",3)'::FRUIT_QTY;

?column?

----------

f

(1row)

ManagingrelateddatawithtriggersServerprogrammingcanalsomeansettingupautomatedactions(triggers),sothatsomeoperationsinthedatabasecausesomeotherthingstohappenaswell.Forexample,youcansetupaprocesswheremakinganofferonsomeitemsisautomaticallyreservedtothembeinginthestocktable.

So,let’screateafruitstocktable,asshownhere:

CREATETABLEfruits_in_stock(

nametextPRIMARYKEY,

in_stockintegerNOTNULL,

reservedintegerNOTNULLDEFAULT0,

CHECK(in_stockbetween0and1000),

CHECK(reserved<=in_stock)

);

TheCHECKconstraintsmakesurethatsomebasicrulesarefollowed:youcan’thavemorethan1000fruitsinstock(they’llprobablygobad),youcan’thaveanegativestock,andyoucan’treservemorethanwhatyouhave.Thefruit_offertablewillcontainthefruitsfromstockwhichareonoffer.Whenweinsertarowinthefruit_offertable.Theofferedamountwillbereservedinthestocktableasshown:

CREATETABLEfruit_offer(

offer_idserialPRIMARYKEY,

recipient_nametext,

offer_datetimestampdefaultcurrent_timestamp,

fruit_nametextREFERENCESfruits_in_stock,

offered_amountinteger

);

TheoffertablehasanIDfortheoffer(soyoucandistinguishbetweenofferslater),recipient,date,offeredfruitname,andofferedamount.

Inordertoautomatethereservationmanagement,youfirstneedaTRIGGERfunction,whichimplementsthemanagementlogic:

CREATEORREPLACEFUNCTIONreserve_stock_on_offer()RETURNStriggerAS$$

BEGIN

IFTG_OP='INSERT'THEN

UPDATEfruits_in_stock

SETreserved=reserved+NEW.offered_amount

WHEREname=NEW.fruit_name;

ELSIFTG_OP='UPDATE'THEN


SETreserved=reserved-OLD.offered_amount

+NEW.offered_amount

WHEREname=NEW.fruit_name;

ELSIFTG_OP='DELETE'THEN



WHEREname=OLD.fruit_name;

ENDIF;

RETURNNEW;

END;

$$LANGUAGEplpgsql;

YouhavetotellPostgreSQLtocallthisfunctioneachandeverytimetheofferrowischanged:

CREATETRIGGERmanage_reserve_stock_on_offer_change

AFTERINSERTORUPDATEORDELETEONfruit_offerFOREACHROWEXECUTE

PROCEDUREreserve_stock_on_offer();

Afterthis,wearereadytotestthefunctionality.First,wewilladdsomefruitstoourstock:

INSERTINTOfruits_in_stockVALUES('APPLE',500);

INSERTINTOfruits_in_stockVALUES('ORANGE',500);

Then,wewillcheckthestock(usingtheexpandeddisplay):

postgres=#\x

Expandeddisplayison.

postgres=#SELECT*FROMfruits_in_stock;

-[RECORD1]----

name|APPLE

in_stock|500

reserved|0

-[RECORD2]----

name|ORANGE

in_stock|500

reserved|0

Next,let’smakeanofferof100applestoBob:

postgres=#INSERTINTO

fruit_offer(recipient_name,fruit_name,offered_amount)

VALUES('Bob','APPLE',100);

INSERT01

postgres=#SELECT*FROMfruit_offer;

-[RECORD1]--+---------------------------

offer_id|1

recipient_name|Bob

offer_date|2013-01-2515:21:15.281579

fruit_name|APPLE

offered_amount|100

Oncheckingthestock,weseethatindeed100applesarereserved,asshowninthefollowingcodesnippet:


-[RECORD1]----

name|ORANGE

in_stock|500

reserved|0

-[RECORD2]----

name|APPLE

in_stock|500

reserved|100

Ifwechangetheofferedamount,thereservedamountalsochanges:

postgres=#UPDATEfruit_offerSEToffered_amount=115WHEREoffer_id=1;

UPDATE1


-[RECORD1]----

name|ORANGE

in_stock|500

reserved|0

-[RECORD2]----

name|APPLE

in_stock|500

reserved|115

Wealsogetsomeextrabenefits.First,becauseoftheconstraintonthestocktable,youcan’tsellthereservedapples:

postgres=#UPDATEfruits_in_stockSETin_stock=100WHEREname='APPLE';

ERROR:newrowforrelation"fruits_in_stock"violatescheckconstraint

"fruits_in_stock_check"

DETAIL:Failingrowcontains(APPLE,100,115).

Moreinterestingly,youalsocan’treservemorethanyouhave,eventhoughtheconstraintsareonanothertable:

postgres=#UPDATEfruit_offerSEToffered_amount=1100WHEREoffer_id=1;

ERROR:newrowforrelation"fruits_in_stock"violatescheckconstraint

"fruits_in_stock_check"

DETAIL:Failingrowcontains(APPLE,500,1100).

CONTEXT:SQLstatement"UPDATEfruits_in_stock


+NEW.offered_amount

WHEREname=NEW.fruit_name"

PL/pgSQLfunctionreserve_stock_on_offer()line8atSQLstatement

Whenyoufinallydeletetheoffer,thereservationisreleased:

postgres=#DELETEFROMfruit_offerWHEREoffer_id=1;

DELETE1


-[RECORD1]----

name|ORANGE

in_stock|500

reserved|0

-[RECORD2]----

name|APPLE

in_stock|500

reserved|0

Inarealsystem,youprobablywillarchivetheoldofferbeforedeletingit.

AuditingchangesIfyouneedtoknowwhodidwhattothedataandwhenitwasdone,onewaytofindoutistologeveryactionthatisperformedinanimportanttable.InPostgreSQL9.3,youcanalsoauditthedatadefinitionlanguage(DDL)changestothedatabaseusingeventtriggers.Wewilllearnmoreaboutthisinthelaterchapters.

Thereareatleasttwoequallyvalidwaystoperformdataauditing:

UsingauditingtriggersAllowingtablestobeaccessedonlythroughfunctionsandauditinginsidethesefunctions

Here,wewilltakealookataminimalnumberofexamplesforboththeapproaches.

First,let’screatethetables:

CREATETABLEsalaries(

emp_nametextPRIMARYKEY,

salaryintegerNOTNULL

);

CREATETABLEsalary_change_log(

changed_bytextDEFAULTCURRENT_USER,

changed_attimestampDEFAULTCURRENT_TIMESTAMP,

salary_optext,

emp_nametext,

old_salaryinteger,

new_salaryinteger

);

REVOKEALLONsalary_change_logFROMPUBLIC;

GRANTALLONsalary_change_logTOmanagers;

Youdon’tgenerallywantyouruserstobeabletochangeauditlogs,soonlygrantthemanagerstherighttoaccessthese.Ifyouplantoletusersaccessthesalarytabledirectly,youshouldputatriggeronitforauditing:

CREATEORREPLACEFUNCTIONlog_salary_change()RETURNStriggerAS$$

BEGIN


INSERTINTOsalary_change_log(salary_op,emp_name,new_salary)

VALUES(TG_OP,NEW.emp_name,NEW.salary);

ELSIFTG_OP='UPDATE'THEN

INSERTINTOsalary_change_log(salary_op,emp_name,old_salary,new_salary)

VALUES(TG_OP,NEW.emp_name,OLD.salary,NEW.salary);

ELSIFTG_OP='DELETE'THEN

INSERTINTOsalary_change_log(salary_op,emp_name,old_salary)

VALUES(TG_OP,NEW.emp_name,OLD.salary);

ENDIF;

RETURNNEW;

END;

$$LANGUAGEplpgsqlSECURITYDEFINER;

CREATETRIGGERaudit_salary_change

AFTERINSERTORUPDATEORDELETEONsalaries

FOREACHROWEXECUTEPROCEDURElog_salary_change();

Now,let’stestoutsomesalarymanagement:

postgres=#INSERTINTOsalariesvalues('Bob',1000);

INSERT01

postgres=#UPDATEsalariesSETsalary=1100WHEREemp_name='Bob';

UPDATE1

postgres=#INSERTINTOsalariesVALUES('Mary',1000);

INSERT01

postgres=#UPDATEsalariesSETsalary=salary+200;

UPDATE2

postgres=#SELECT*FROMsalaries;

-[RECORD1]--

emp_name|Bob

salary|1300

-[RECORD2]--

emp_name|Mary

salary|1200

Eachoneofthesechangesissavedintothesalarychangelogtableforauditingpurposes:

postgres=#SELECT*FROMsalary_change_log;

-[RECORD1]--------------------------

changed_by|frank

changed_at|2012-01-2515:44:43.311299

salary_op|INSERT

emp_name|Bob

old_salary|

new_salary|1000

-[RECORD2]--------------------------

changed_by|frank

changed_at|2012-01-2515:44:43.313405

salary_op|UPDATE

emp_name|Bob

old_salary|1000

new_salary|1100

-[RECORD3]--------------------------

changed_by|frank

changed_at|2012-01-2515:44:43.314208

salary_op|INSERT

emp_name|Mary

old_salary|

new_salary|1000

-[RECORD4]--------------------------

changed_by|frank

changed_at|2012-01-2515:44:43.314903

salary_op|UPDATE

emp_name|Bob

old_salary|1100

new_salary|1300

-[RECORD5]--------------------------

changed_by|frank

changed_at|2012-01-2515:44:43.314903

salary_op|UPDATE

emp_name|Mary

old_salary|1000

new_salary|1200

Ontheotherhand,youmaynotwantanybodytohavedirectaccesstothesalarytable,inwhichcaseyoucanperformtheREVOKEcommand.ThefollowingcommandwillrevokeallprivilegesfromPUBLIC:

REVOKEALLONsalariesFROMPUBLIC;

Also,giveusersaccesstoonlytwofunctions:thefirstfunctionisforanyusertakingalookatsalariesandtheotherfunctioncanbeusedtochangesalaries,whichisavailableonlytomanagers.

ThefunctionswillhavealltheaccesstotheunderlyingtablesbecausetheyaredeclaredasSECURITYDEFINER,whichmeansthattheyrunwiththeprivilegesoftheuserwhocreatedthem.

Thisishowthesalarylookupfunctionwilllook:

CREATEORREPLACEFUNCTIONget_salary(text)

RETURNSinteger

AS$$

—ifyoulookatotherpeople'ssalaries,itgetslogged


SELECT'SELECT',emp_name,salary

FROMsalaries

WHEREupper(emp_name)=upper($1)

ANDupper(emp_name)!=upper(CURRENT_USER);

—don'tlogselectofownsalary

—returntherequestedsalary

SELECTsalaryFROMsalariesWHEREupper(emp_name)=upper($1);

$$LANGUAGESQLSECURITYDEFINER;

Noticethatweimplementedasoft-securityapproach,whereyoucanlookupotherpeople’ssalaries,butyouhavetodoitresponsibly,thatis,onlywhenyouneedto,asyourmanagerwillknowthatyouhavechecked.

Theset_salary()functionabstractsawaytheneedtocheckwhethertheuserexists;iftheuserdoesnotexist,itiscreated.Settingsomeone’ssalaryto0willremovehimorherfromthesalarytable.Thus,theinterfaceissimplifiedtoalargeextent,andtheclientapplicationofthesefunctionsneedstoknow,anddo,less:

CREATEORREPLACEFUNCTIONset_salary(i_emp_nametext,i_salaryint)

RETURNSTEXTAS$$

DECLARE

old_salaryinteger;

BEGIN

SELECTsalaryINTOold_salary

FROMsalaries

WHEREupper(emp_name)=upper(i_emp_name);

IFNOTFOUNDTHEN

INSERTINTOsalariesVALUES(i_emp_name,i_salary);


VALUES('INSERT',i_emp_name,i_salary);

RETURN'INSERTEDUSER'||i_emp_name;

ELSIFi_salary>0THEN

UPDATEsalaries

SETsalary=i_salary

WHEREupper(emp_name)=upper(i_emp_name);

INSERTINTOsalary_change_log

(salary_op,emp_name,old_salary,new_salary)

VALUES('UPDATE',i_emp_name,old_salary,i_salary);

RETURN'UPDATEDUSER'||i_emp_name;

ELSE—salarysetto0

DELETEFROMsalariesWHEREupper(emp_name)=upper(i_emp_name);

INSERTINTOsalary_change_log(salary_op,emp_name,old_salary)

VALUES('DELETE',i_emp_name,old_salary);

RETURN'DELETEDUSER'||i_emp_name;

ENDIF;

END;


Now,droptheaudittrigger(otherwisethechangeswillbeloggedtwice)andtestthenewfunctionality:

postgres=#DROPTRIGGERaudit_salary_changeONsalaries;

DROPTRIGGER

postgres=#

postgres=#SELECTset_salary('Fred',750);

-[RECORD1]------------------

set_salary|INSERTEDUSERFred

postgres=#SELECTset_salary('frank',100);

-[RECORD1]-------------------

set_salary|INSERTEDUSERfrank


-[RECORD1]---

emp_name|Bob

salary|1300

-[RECORD2]---

emp_name|Mary

salary|1200

-[RECORD3]---

emp_name|Fred

salary|750

-[RECORD4]---

emp_name|frank

salary|100

postgres=#SELECTset_salary('mary',0);

-[RECORD1]-----------------

set_salary|DELETEDUSERmary


-[RECORD1]---

emp_name|Bob

salary|1300

-[RECORD2]---

emp_name|Fred

salary|750

-[RECORD3]---

emp_name|frank

salary|100

postgres=#SELECT*FROMsalary_change_log;

...

-[RECORD6]--------------------------

changed_by|gsmith

changed_at|2013-01-2515:57:49.057592

salary_op|INSERT

emp_name|Fred

old_salary|

new_salary|750

-[RECORD7]--------------------------

changed_by|gsmith

changed_at|2013-01-2515:57:49.062456

salary_op|INSERT

emp_name|frank

old_salary|

new_salary|100

-[RECORD8]--------------------------

changed_by|gsmith

changed_at|2013-01-2515:57:49.064337

salary_op|DELETE

emp_name|mary

old_salary|1200

new_salary|

DatacleaningIntheprecedingcode,wenoticethatemployeenamesdon’thaveconsistentcases.Itwillbeeasytoenforceconsistencybyaddingaconstraint,asshownhere:

CHECK(emp_name=upper(emp_name))

However,itisevenbettertojustmakesurethatthenameisstoredasuppercase,andthesimplestwaytodothisisbyusingtrigger:

CREATEORREPLACEFUNCTIONuppercase_name()

RETURNStriggerAS$$

BEGIN

NEW.emp_name=upper(NEW.emp_name);

RETURNNEW;

END;

$$LANGUAGEplpgsql;

CREATETRIGGERuppercase_emp_name

BEFOREINSERTORUPDATEORDELETEONsalaries

FOREACHROWEXECUTEPROCEDUREuppercase_name();

Thenextset_salary()callforanewemployeewillnowinsertemp_nameinuppercase:

postgres=#SELECTset_salary('arnold',80);

-[RECORD1]-------------------

set_salary|INSERTEDUSERarnold

Astheuppercasinghappensinsideatrigger,thefunction’sresponsestillshowsalowercasename,butinthedatabase,itisuppercased:


-[RECORD1]---

emp_name|Bob

salary|1300

-[RECORD2]---

emp_name|Fred

salary|750

-[RECORD3]---

emp_name|Frank

salary|100

-[RECORD4]---

emp_name|ARNOLD

salary|80

Afterfixingtheexistingmixed-caseemployeenames,wecanmakesurethatallemployeenameswillbeuppercasedinthefuturebyaddingaconstraint:

postgres=#updatesalariessetemp_name=upper(emp_name)wherenot

emp_name=upper(emp_name);

UPDATE3

postgres=#altertablesalariesaddconstraint

emp_name_must_be_uppercasepostgresCHECK(emp_name=upper(emp_name));

ALTERTABLE

Ifthisbehaviorisneededinmoreplaces,itwillmakesensetodefineanewtype–sayu_text,whichisalwaysstoredasuppercase.YouwilllearnmoreaboutthisapproachinChapter14,PostgreSQLasExtensibleRDBMS.

CustomsortordersThelastexampleinthischapter,isaboutusingfunctionsfordifferentwaysofsorting.

Saywearegivenatasktosortwordsbytheirvowelsonly,andinadditiontothis,tomakethelastvowelthemostsignificantonewhensorting.Whilethistaskmayseemreallycomplicatedatfirst,itcanbeeasilysolvedwithfunctions:

CREATEORREPLACEFUNCTIONreversed_vowels(wordtext)

RETURNStextAS$$

vowels=[cforcinword.lower()ifcin'aeiou']

vowels.reverse()

return''.join(vowels)

$$LANGUAGEplpythonuIMMUTABLE;

postgres=#selectword,reversed_vowels(word)fromwordsorderby

reversed_vowels(word);

word|reversed_vowels

-------------+-----------------

Abracadabra|aaaaa

Great|ae

Barter|ea

Revolver|eoe

(4rows)

NoteBeforeperformingthiscode,pleasemakesureyouhavePython2.xinstalled.WewilldiscussPL/Pythoninmuchdetailinthelaterchaptersofthisbook.

Thebestpartisthatyoucanuseyournewfunctioninanindexdefinition:

postgres=#CREATEINDEXreversed_vowels_indexONwords

(reversed_vowels(word));

CREATEINDEX

Thesystemwillautomaticallyusethisindexwheneverthereversed_vowels(word)functionisusedintheWHEREorORDERBYclause.

ProgrammingbestpracticesDevelopingapplicationsoftwareiscomplicated.Someoftheapproachesthathelpmanagethiscomplexityaresopopularthattheyhavebeengivensimpleacronymsthatcanberemembered.Next,we’llintroducesomeoftheseprinciplesandshowyouhowserverprogramminghelpsmakethemeasiertofollow.

KISS–keepitsimplestupidOneofthemaintechniquestosuccessfulprogrammingiswritingsimplecode.Thatis,writingcodethatyoucaneasilyunderstand3yearsfromnowandthatotherscanunderstandaswell.Itisnotalwaysachievable,butitalmostalwaysmakessensetowriteyourcodeinthesimplestwaypossible.Youcanrewritepartsofitlaterforvariousreasonssuchasspeed,codecompactness,toshowoffhowcleveryouare,andsoon.However,alwayswritethecodeinasimplewayfirst,sothatyoucanbeabsolutelysurethatitdoeswhatyouwant.Notonlydoyougetworkingonthecodequickly,butyoualsohavesomethingtocomparetowhenyoutrymoreadvancedwaystodothesamething.

Remember,debuggingisharderthanwritingcode;so,ifyouwritethecodeinthemostcomplexwayyoucan,youwillhaveareallyhardtimedebuggingit.

Itisofteneasiertowriteasetreturningfunctioninsteadofacomplexquery.Yes,itwillprobablyrunslowerthanthesamethingimplementedasasinglecomplexquery,duetothefactthattheoptimizercandoverylittletothecodewrittenasfunctions,butthespeedmaybesufficientforyourneeds.Ifmorespeedisrequired,it’sverylikelytorefactorthecodepiecebypiece,joiningpartsofthefunctionintolargerquerieswheretheoptimizerhasabetterchanceofdiscoveringbetterqueryplansuntiltheperformanceisacceptableagain.

Rememberthatmostofthetime,youdon’tneedtheabsolutelyfastestcode.Foryourclientsorbosses,thebestcodeistheonethatdoesthejobwellandarrivesontime.

DRY–don’trepeatyourselfThisprinciplemeansyoushouldimplementanypieceofbusinesslogicjustonceandputthecodefordoingitintherightplace.

Thismaybehardsometimes;forexample,youwanttodosomechecksonyourwebformsinthebrowser,butstilldothefinalchecksinthedatabase.However,asageneralguideline,itisverymuchvalid.

Serverprogramminghelpsalothere.Ifyourdatamanipulationcodeisinthedatabasenearthedata,allthedatausershaveeasyaccesstoit,andyouwillnotneedtomanageasimilarcodeinaC++Windowsprogram,twoPHPwebsites,andabunchofPythonscriptsdoingnightlymanagementtasks.Ifanyofthemneedtodothisthingtoacustomer’stable,theyjustcall:

SELECT*FROMdo_this_thing_to_customers(arg1,arg2,arg3);

That’sit!

Ifthelogicbehindthefunctionneedstobechanged,youjustchangethefunctionwithnodowntimeandnocomplicatedorchestrationofpushingdatabasequeryupdatestoseveralclients.Oncethefunctionischangedinthedatabase,itischangedforalltheusers.

YAGNI–youain’tgonnaneeditInotherwords,don’tdomorethanyouabsolutelyneedto.

Ifyouhaveacreepyfeelingthatyourclientisnotyetwellawareofhowthefinaldatabasewilllookorwhatitwilldo,it’shelpfultoresisttheurgetodesigneverythingintothedatabase.Amuchbetterwayistodoaminimalimplementationthatsatisfiesthecurrentspecifications,butdoitwithextensibilityinmind.Itisveryeasyto“paintyourselfintoacorner”whenimplementingabigspecificationwithlargeimaginaryparts.

Ifyouorganizeyouraccesstothedatabasethroughfunctions,itisoftenpossibletodoevenlargerewritesofbusinesslogicwithouttouchingthefrontendapplicationcode.YourapplicationstillperformsSELECT*FROMdo_this_thing_to_customers(arg1,arg2,arg3),evenafteryouhaverewrittenthefunctionfivetimesandchangedthewholetablestructuretwice.

SOA–service-orientedarchitectureUsually,whenyouheartheacronymSOA,itwillbefromenterprisesoftwarepeopletryingtosellyouacomplexsetofSOAPservices.ButtheessenceofSOAistoorganizeyoursoftwareplatformasasetofservicesthatclients,andotherservices,callinordertoperformcertainwell-definedatomictasks,asfollows:

Checkingauser’spasswordandcredentialsPresentinghim/herwithalistofhis/herfavoritewebsitesSellinghim/heranewreddogcollarwithacomplementarymembershipinthered-collareddogclub

TheseservicescanbeimplementedasSOAPcallswithcorrespondingWSDLdefinitionsandJavaserverswithservletcontainers,aswellasacomplexmanagementinfrastructure.TheycanalsobeasetofPostgreSQLfunctions,takingasetofargumentsandreturningasetofvalues.Iftheargumentsorreturnvaluesarecomplex,theycanbepassedasXMLorJSON,butasimplesetofstandardPostgreSQLdatatypesisoftenenough.InChapter10,ScalingYourDatabasewithPL/Proxy,youwilllearnhowtomakesuchaPostgreSQL-basedSOAserviceinfinitelyscalable.

TypeextensibilitySomeoftheprecedingtechniquesareavailableinotherdatabases,butPostgreSQL’sextensibilitydoesnotstophere.InPostgreSQL,youcanjustwriteUDFsinanyofthemostpopularscriptinglanguages.Youcanalsodefineyourowntypes,notjustdomains,whicharestandardtypeswithsomeextraconstraintsattached,andnewfull-fledgedtypestoo.

Forexample,aDutchcompany,MGRID,hasdevelopedavaluewithunitsetofdatatypes,sothatyoucandivide10kmby0.2hoursandgettheresultin50km/h.Ofcourse,youcanalsocastthesameresulttometerspersecondoranyotherunitofspeed.Andyes,youcangetthisasafractionofc—thespeedoflight.

Thiskindoffunctionalityneedsboththetypesandoverloadedoperands,whichknowthatifyoudividedistancebytime,thentheresultisspeed.Youwillalsoneeduser-definedcasts,whichareautomaticallyormanually-invokedconversionfunctionsbetweentypes.

MGRIDdevelopedthisforuseinmedicalapplications,wherethecostofanerrorcanbehigh—thedifferencebetween10mland10cccanbevital.However,usingasimilarsystemmightalsohaveavertedmanyotherdisasters,wherewrongunitsendedupproducingbadcomputationresults.Iftheamountisalwaysaccompaniedbytheunit,thepossibilityforthesekindsoferrorsisdiminished.Youcanalsoaddyourownindexmethodifyouhavesomeprogrammingskillsandyourproblemdomainisnotwellservedbytheexistingindexes.ThereisalreadyarespectablesetofindextypesincludedinthecorePostgreSQL,aswellasseveralothersthataredevelopedoutsidethecore.

ThelatestindexmethodthatbecameofficiallyincludedinPostgreSQLisknearestneighbor(KNN)—acleverindex,whichcanreturnKrowsorderedbytheirdistancefromthedesiredsearchtarget.OneuseofKNNisinfuzzytextsearch,wherethiscanbeusedtorankfull-textsearchresultsbyhowwelltheymatchthesearchterms.BeforeKNN,thiskindofthingwasdonebyqueryingalltherowswhichmatchedevenslightly,thensortingallthesebythedistancefunction,andreturningKtoprowsasthefinalstep.

IfdoneusingtheKNNindex,theindexaccesscanstartreturningtherowsinthedesiredorder;so,asimpleLIMITKfunctionwillreturntheKtopmatches.

TheKNNindexcanalsobeusedforrealdistances,forexample,answeringtherequest“Givemethe10nearestpizzaplacestoCentralStation.”

Asyousaw,indextypesaredifferentfromthedatatypestheyindex.Anotherexample,isthesameGeneralInvertedIndex(GIN)canbeusedforfull-textsearches(togetherwithstemmers,thesauri,andothertext-processingstuff),aswellasforindexingelementsofintegerarrays.

CachingYetanotherplacewhereserver-sideprogrammingcanbeusedistocachevalues,whichareexpensivetocompute.Thefollowingisthebasicpatternhere:

1. Checkwhetherthevalueiscached.2. Ifitisn’t,orthevalueistooold,computeandcacheit.3. Returnthecachedvalue.

Forexample,calculatingthesalesforacompanyistheperfectitemtocache.Perhaps,alargeretailcompanyhas1,000storeswithpotentiallymillionsofindividualsales’transactionsperday.Ifthecorporateheadquartersislookingforsales’trends,itismuchmoreefficientifthedailysalesnumbersareprecalculatedatthestorelevelinsteadofsummingupmillionsofdailytransactions.

Ifthevalueissimple,suchaslookingupauser’sinformationfromasingletablebasedontheuserID,youdon’tneedtodoanything.ThevaluegetscachedinPostgreSQL’sinternalpagecache,andalllookupstoitaresofastthatevenonaveryfastnetwork,mostofthetimeisspentdoingthelookupsinthenetworkandnotintheactuallookup.Insuchacase,gettingdatafromaPostgreSQLdatabaseisasfastasgettingitfromanyotherin-memorycache(suchasmemcached)butwithoutanyextraoverheadinmanagingthecache.

Anotherusecaseofcachingistoimplementmaterializedviews.Theseareviewsthatareprecomputedonlywhenrequired,noteverytimeoneselectsdatafromtheview.SomeSQLdatabaseshavematerializedviewsasseparatedatabaseobjects,butinthePostgreSQLversionspriorto9.3,youhavetodoityourselfusingotherdatabasefeaturestoautomatethewholeprocess.

Wrappingup–whyprogramintheserver?Themainadvantagesofdoingmostdatamanipulationcodeontheserver-sidearestatedinthefollowingsections.

PerformanceDoingthecomputationnearthedataisalmostalwaysaperformancewin,asthelatenciestogetthedataareminimal.Inatypicaldata-intensivecomputation,mostofthetimeisspentingettingthedata.Therefore,makingdataaccessinsidethecomputationfasteristhebestwaytomakethewholethingfast.Onmylaptop,ittakes2.2mstoqueryonerandomrowfroma1,000,000-rowdatabaseintotheclient,butittakesonly0.12mstogetthedatainsidethedatabase.Thisis20timesfasterandinsidethesamemachineoverUnixsockets.Thedifferencecanbebiggerifthereisanetworkconnectionbetweentheclientandtheserver.

Asmallreal-wordstory:

Afriendofminewascalledtohelpalargecompany(I’msureallofyouknowit,butIcan’ttellyouwhichone)inordertomakeitse-mailsendingapplicationfaster.Theyhadimplementedtheire-mailgenerationsystemwithallthelatestJavaEEtechnologies:first,gettingthedatafromthedatabase,passingthedataaroundbetweenservices,andserializinganddeserializingitseveraltimesbeforefinallydoingXSLTtransformationonthedatatoproducethee-mailtext.Theendresultbeingthatitproducedonlyafewhundrede-mailspersecond,andtheywerefallingbehindwiththeirresponses.

WhenherewrotetheprocesstouseaPL/Perlfunctioninsidethedatabasetoformatthedataandthequeryreturnedalreadyfully-formattede-mails,itsuddenlystartedspewingouttensofthousandsofe-mailspersecondandtheyhadtoaddasecondcopyofthesentmailtoactuallybeabletosendthemout.

EaseofmaintenanceIfallthedatamanipulationcodeisinadatabase,eitherasdatabasefunctionsorviews,theactualupgradeprocessbecomesveryeasy.AllthatisneededistorunaDDLscriptthatredefinesthefunctions;alltheclientsautomaticallyusethenewcodewithnodowntimeandnocomplicatedcoordinationbetweenseveralfrontendsystemsandteams.

ImprovedproductivityServer-sidefunctionsareperhapsthebestwaytoachievecodereuse.Anyclientapplicationwritteninanylanguageorframeworkcanmakeuseoftheserver-sidefunctions,ensuringmaximumreuseinallenvironments.

SimplewaystotightensecurityIfalltheaccessforsomepossiblyinsecureserversgoesthroughfunctions,thedatabaseuseroftheseserverscanonlybegrantedaccesstotheneededfunctionsandnothingelse.Theycan’tseethetabledataoreventhefactthatthesetablesexist.So,eveniftheserveriscompromised,allitcandoiscontinuetocallthesamefunctions.Also,thereisnopossibilityofstealingpasswords,e-mails,orothersensitiveinformationbyissuingitsownqueriessuchasSELECT*FROMusers;andgettingallthedatathereisinthedatabase.

Also,themostimportantthingisthatprogramminginaserverisfun!

SummaryProgramminginsidethedatabaseserverisnotalwaysthefirstthingthatcomestomindtomanydevelopers,butitsuniqueplacementinsidetheapplicationstackgivesitsomepowerfuladvantages.Yourapplicationcanbefaster,moresecure,andmoremaintainablebypushinglogicintothedatabase.Withserver-sideprogramminginPostgreSQL,youcansecureyourdatausingfunctions,auditaccesstoyourdataandstructuralchangesusingtriggers,andimproveproductivitybyachievingcodereuse.Also,youcanenrichyourdatausingcustomdatatypes,analyzeyourdatausingcustomoperators,andextendthecapabilitiesofthedatabasebydynamicallyloadingnewfunctions.

ThisisjustthestartofwhatyoucandoinsidePostgreSQL.Throughouttherestofthisbook,youwilllearnmanyotherwaystowritepowerfulapplicationsbyprogramminginsidePostgreSQL.

Chapter2.ServerProgrammingEnvironmentsYou’vehadachancetogetacquaintedwiththegeneralideaofusingPostgreSQL,butnowwearegoingtoanswerthequestionofwhyanyonewillchoosePostgreSQLasadevelopmentplatform.AsmuchasI’dliketobelievethatit’saneasydecisionforeveryone,it’snot.

Forstarters,let’sgetridoftheoptimisticideathatyouchooseadatabaseplatformfortechnicalreasons.Sure,weallliketothinkthatweareobjective,andwebaseourdecisionsonapreponderanceofthetechnicalevidence.Thispreponderanceofevidencethenindicateswhichfeaturesareavailableandrelevanttoourapplication.Wewillthenproceedtomakeaweightedchoiceinfavorofthemostadvantageousplatform,anduseabalanceoftheevidencestocreateworkaroundsandalternativeswhereourchoicefallsshort.Thefactisthatwedon’treallyunderstandalltherequirementsoftheapplicationuntilwearehalfwaythroughthedevelopmentcycle.Herearesomereasonswhy:

Wedon’tknowhowtheapplicationwillevolveovertime.Manystart-upspivotfromtheirinitialideaasthemarkettellsthemtochange.Wedon’tknowhowmanyuserstherewillreallybeuntilwehavesomeregistrationsandcanbegintomeasurethecurve.Wedon’trealizehowimportantaparticularfeaturecanbeuntilwegetuserfeedback.Thetruthis,thatwedon’treallyknowmuchaboutthelong-termneedsoftheapplicationuntilwe’rewritingversion2ormaybeevenversion3.

Thatis,unlessyou’reoneofthefortunatefewwhohasaResearchandDevelopmentdepartmentthatwritesthealphaversion,throwsitoutthewindow,andthenasksyoutowritethenextversionbasedonthelessonslearned.Eventhen,youreallydon’tknowwhattheusagepatternsaregoingtobeoncetheapplicationisdeployed.

WhatwegenerallyseeinthePostgreSQLcommunity—whennewusersstartaskingquestions—ispeoplenotlookingtomakeadecision,butratherpeoplewhohavealreadymadeadecision.Inmostcases,theyarelookingfortechnicaljustificationforanexistingplanofaction.Thedecisionhasalreadybeenpassed.WhatIamgoingtowriteaboutinthischapterisnotaTPCbenchmark,norisitabouttherelativemeritsofPostgreSQLfunctionsversusstoredprocedures.Frankly,nobodyreallycaresaboutthesethingsuntiltheyhavealreadymadeachoiceandwanttojustifyit.

ThischaptercontainstheguidethatIwishsomeonehadwrittenformewhenIchosetousePostgreSQLbackin1998.

CostofacquisitionOneofbiggestthefactorsthatdecideswhichtechnologyisusedintheapplicationstackisthecostofacquisition.I’veseenmanyapplicationarchitecturesdrawnonawhiteboardwherethetechnicalteamwasembarrassedtoshowthem,buttheyjustifiedthedesignbytryingtokeepsoftwarelicensingcostsdown.Whenitcomestothedatabaseenvironment,theusualsuspectsareOracle,SQLServer,MySQL,andPostgreSQL.Oracle,thedominantplayerinthedatabasespace,isalsothemostcostly.Atthelowend,OracledoeshavereasonablypricedofferingsandevenafreeExpressEdition,buttheyarelimited.Mostpeoplehaveneedsbeyondthelow-pricedofferingsandfallintotheenterprisesalesmachineofOracle.Thisusuallyresultsinahigh-pricequotethatmakesyourCFOfalloutofhis/herchair,andyou’rebacktodesigningyoursolutioninordertokeepyourlicensingcostsdown.

ThencomesMicrosoftSQLServer.Thisisyourfirstreasonablyviableoption.ThepricingislistedontheMicrosoftwebsite.Iwillnotreproduceitherebecausethepricingscheduleistoovolatileforabookthatwillremaininprintforalongertime.Nonetheless,anexperiencedthumbvalueofthepurchasecostforSQLServerwillgetyourunningwithaweb-capablemodelforabout$5,000.Thisdoesnotincludeaservicecontract.Inthegrandschemeofdevelopmentcosts,thisisreasonableandnottoohighofabarriertoenter.

Then,wehavetheopensourceofferingssuchasMySQLandPostgreSQL.Theycostnothingandtheservicecontractscost—waitforit—nothing.Thisisaveryhardcostofacquisitiontobeat.

Remember,inthebeginningofthechapter,whenIwastalkingaboutallthethingsthatyoudon’tknowwhentheprojectstarts?Here’swheretherealwincomesin.Youcanaffordtofail.

There,Isaidit!

Lowcostofacquisitionisasynonymforlowcostoffailure.Whenweaddupalloftheunknownsfortheproject,wefindoutthatwehaveafairlygoodchancethatthefirstiterationwillnotmeetthemarketneeds,andweneedtofindawaytojettisonitquicklywithoutlong-termcontractsandtheadditionalcostsofspinningupanewproject.

Thisallowstheprojectmanagertomoveontothenextversionusinglessonslearnedfromtheconsumerafterthefirstversion.Hopefully,thislessoninuseracceptancewillcomeataverylowcostandtheprojectwillthenbegintothriveinthefollowingversions.Don’tletthesuccessoftheprojecthangongettingthefirstversionperfect.Youwon’t!

AvailabilityofdevelopersThishasbeenoneofthemosthilariouspartsofmydevelopmentlife.IrecentlyrecommendedalocalcompanytousePostgreSQLforareportingsystem.ThecompanyinquestionwantedtoknowthatiftheychosePostgreSQL,wouldanyoneonstaffbeabletomaintainit.So,IbegantointerviewthedeveloperstofindoutabouttheirexperienceswithPostgreSQL.

Me:DoyouhaveanyexperiencewithPostgreSQL?

Developer1:Yes,Iuseditatthelastjobforaproductfulfillmentproject,butIdon’tthinkmanypeoplehavethatexperience.WeshouldprobablysticktousingMySQL.

Me:DoyouhaveanyexperiencewithPostgreSQL?

Developer2:Yes,Iuseditatthelastjobforareportingproject,butIdon’tthinkmanypeoplehavethatexperience.WeshouldprobablysticktousingMySQL.

Afterinterviewingallsevendevelopersthatwereinfluentialontheproject,Ifoundthattheonlypersonwithouthands-onexperiencewithPostgreSQLwastheprojectmanager.Sincetheprojectmanagerdidn’texpecttohaveanytechnicalinvolvementintheproject,heapprovedtheselectionofPostgreSQL.

PostgreSQLisoneofthedirtylittlesecretsofwebdevelopers.Theyhaveaboutthesameleveloffamiliaritywithitastheydowithencryptionandsecurity.Becauseonlyadvanceduserswilluseit,theyhaveageneralgeekrequirementtolookintoitandpresumethateveryoneelseistooinexperiencedtodothesame.Everyoneistryingto“dumbitdown”fortheotherguy.Theyconsidertheirownuseofthetoolsathand(MySQL)asacrificethattheyarewillingtomakeinordertohelpthelessexperiencedpersondownthehall.Comically,thepersondownthehallthinksthathe’s/she’smakingthesamesacrificeforeveryoneelse.

TipLessonlearned

Quitmakingchoicesfortheotherguy.He/sheisjustasexperienced(andintelligent)asyouare,orhe/shemightjustwanttheopportunitytoadvancehis/herskills.

LicensingAbout2monthsafterOracleboughtMySQL,theyannouncedaplanthatdividedthedevelopmentintotwocamps:aMySQLcommunityeditionandaprofessionalversion.Thecommunityeditionwouldnolongergainanynewfeatures,andtheprofessionalversionwouldbecomeacommercialproduct.

Therewasavastandthunderoussuckingsoundintheopensourcecommunity,astheythrashedwildlyabouttofindanewplatformforFreeandOpenSourceSoftware(FOSS)development.

Oracleimmediately(inabout2weeks)countermandedtheorderanddeclaredthatthingswillstayastheywerefortheindefinitefuture.Thosewithshortmemories,forgivinghearts,orwhojustweren’tpayingattentionwentonabouttheirbusiness.ManyotheropensourceprojectseitherswitchedtoPostgreSQLorsuddenlygrewPostgreSQLdatabasesupport.

Today,wehaveMySQLandMySQLEnterpriseEdition.Ifyouwantbackup,highavailability,enterprisescalability,andtheMySQLEnterpriseMonitor,younowhavetoponyupsomedough.Capitalismisfine,andcorporationshavearighttochargemoneyfortheirservicesandproductsinordertoexist.Butwhyshouldyou,asaprojectmanagerordeveloper,havetopayforsomethingthatyoucangetforfree?

Licensingisallaboutcontinuedproductavailabilityanddistribution.ThePostgreSQLlicensingmodelspecificallystatesthatyoucanhavethesourcecode,doanythingwithit,redistributeithoweveryoujollywellplease,andtheserightsextendindefinitely.Trytogetthisdealwithacommercialvendor.

Asacorporatedeveloper,PostgreSQLwinsthelegalbattleforriskmanagementhandsdown.Ihaveheardtheargument“IwanttogowithacommercialvendorifIneedsomeonetosue.”Iwillencourageanyonewhoconsidersitagoodargumenttodoalittleresearchabouthowoftenthesevendorshavebeensued,howoftenthosesuitsweresuccessful,andwhatthecostofcourtwasforthatsuccess.Ithinkyou’llfindthattheonlyviableoptionisnottohavethebattle.

PredictabilityThissectioncouldjustaswellhavebeentitledstandardscompliance,butIdecidedagainstitbecausethebenefitsofstandardscomplianceincorporateprojectsarenotobvious.Thelimitationsofthecommondatabasesarewell-documented,andIwillshowyouafewwebsitesinamomentwhereyoucanmakeacomparisonofwhohasthemostunintendedbehavior.Iwillencourageyoutoreadthefollowingmaterialwhilethinkingaboutthequestion,“Whichmethodoffeaturedevelopmentismostlikelytomakemyapplicationbreakinthefuture?”:

http://www.sql-info.de/postgresql/postgres-gotchas.htmlhttp://www.sql-info.de/mysql/gotchas.html

NoteSpoileralert:

Astricteradherencetostandardscomesatthecostofnotallowingambiguousbehavior.Notallowingambiguousbehaviormakesthedeveloper’slifemoredifficult.Makingthedeveloper’slifemoredifficultensuresthattheinterpretationofthecommandsthatthedevelopergiveswillnotchangelater,breakingtheapplication.

Justhowlazycanyouaffordtobe?I’mnotsurehowtomeasurethis.PostgreSQLisavailableforno-costfuturepredictability,soIdon’thavetoanswerthequestion.

Sure,PostgreSQLalsohassomebugslisted.However,changestothedatabasecorehaveatendencytomaketheengineworklikethedocumentationsaysitdoes,notlikethedocumentationshouldhavesaid.PostgreSQLdevelopersdon’thavetosay,“Oops,Ididn’tthinkofthat,”veryoften.Whentheydo,PostgreSQLjustbecomesmorestandardscompliant.

http://www.sql-info.de/postgresql/postgres-gotchas.html

http://www.sql-info.de/mysql/gotchas.html

CommunityOracleandSQLServerdon’thaveacommunity.PleaseunderstandwhenIsaythat,Imeanthatthechancethatyouwillgettotalktoadeveloperofthecoredatabaseisaboutthesameasyourchanceofwinningthelottery.Bythetimeyoudo,it’sprobablybecauseyoufoundabugsoheinousthatitcouldn’tbeignoredandtheonlypersonwhocanunderstandyourreportistheguywhowrotethecodeinquestion.Theyhavepaidtechnicalsupportandthissupporthasproveninmyexperiencetobegenerallycompetent,butnotstellar.IhavehadtoworkaroundtheproblemthatIoriginallyrequestedhelpwithabout40percentofthetime.

ComparethistoMySQLandPostgreSQL,wherejustaboutanybodycanspeaktojustaboutanybodyelsealldaylong.ManyofthecoredevelopersofboththeplatformscanbefoundonIRC,metatconventions,contactedforcontractdevelopmentwork,andforthemostpart,bribedremarkablyeasilywithbeer(hint,hint,wink,wink,nudge,nudge).

Theyareactivelyconcernedaboutthehealthoftheoverallcommunityandwillanswerjustaboutanykindofquestionyouask,evenifthequestionhasaverytenuousrelationshiptodatabasedevelopment.Mypersonalexperience,isthatthePostgreSQLteamhasmorecoredevelopersreadilyavailablethanMySQL.Theyarealsomorepersonallyavailableatconventionsandmeetings.

DidImentiontheylikebeer?

ProcedurallanguagesSQLServerallowsyoutocreateadynamiclinklibrary(DLL)inanylanguagethatproducestheCommonLanguageRuntime(CLR).TheseDLLsmustbeloadedintotheserveratboottime.Tocreateaprocedureatruntimeandhaveitimmediatelyavailable,theonlychoiceisthebuilt-inSQLdialect,TransactSQL(TSQL).

MySQLhasafeaturecalledplugins.Oneofthelegalplugintypesisaprocedurallanguage.SeverallanguageshavebeentooledtoworkwithMySQLviathepluginsystem,includingmostofthepopularonessuchasPHPandPython.Thesefunctionscannotbeusedforstoredproceduresortriggers,buttheycanbeinvokedfromthecommonSQLstatements.Fortherest,youarestuckwiththebuilt-inSQL.

PostgreSQLhasfullsupportforadditionalprocedurallanguages,whichcanbeusedtocreateanylegalentityinthedatabasethatcanbecreatedwithPL/pgSQL.Thelanguagecanbeadded(orremoved)fromarunningversionofPostgreSQLandanyfunctiondefinedusingthislanguagecanalsobecreatedordroppedwhilesupportforadditionalprocedurallanguages,whichcanbeusedtocreateanylegalentityinthedatabasethatcanbecreatedwithPL/pgSQL.Thelanguagecanbeadded(orremoved)fromarunningversionofPostgreSQLandanyfunctiondefinedusingthislanguagecanalsobecreatedordroppedwhilePostgreSQLisrunning.

TheselanguageshavefullaccesstoPostgreSQL’sinternalfunctionsandtoallthedataentitiesthatthecallinguserhaspermissionfor.InadditiontohavingaccesstointernalPostgreSQLfunctions,entities,anddatastructures,somefunctions(inuntrustedlanguages)canalsoaccessexternalservices,createordeletefilesanddirectories,orsende-mailsandinvokeexternalprocesses.Wewilldiscusstrustedanduntrustedlanguagesinlaterchapters.

ManyofthesepluginlanguageextensionsareavailableforPostgreSQL.IhaveusedtheextensionsforPHP,Python,Bash,andPL/pgSQL.Yes,thismeansthatthestandardlanguageforPostgreSQLisalsoinstalledandmanagedusingthesameextensionsystemasanyotherlanguage.

ThisbringsustothepointthatwehavemoredevelopersavailableforPostgreSQLthanyoumighthaveoriginallythought.Softwaredevelopersarenotrequiredtolearnanewdevelopmentlanguageinordertowritestoredprocedures.TheycanextendPostgreSQLwithalanguageoftheirchoiceandcontinuetocodeinthemannerandworkflowthattheychoose.

TipLessonlearned

Therearenosecond-classcitizensinthePostgreSQLdevelopmentcommunity.Anyonecancodein(almost)anylanguagetheychoose.

Third-partytoolsAfrequentpointofcomparisonamongthedatabaseplatformsisthenumberofthird-partyapplicationsavailable.I’mnotsosurethatthetotalnumbermatters,asmuchastheexistenceoftheapplicationsyouactuallyneed.

Tothisend,thefollowingisalistoftheproductsthatIhaveusedextensivelywithPostgreSQL:

Pentahodataintegration(kettle):ThisisanoutstandingExtract,TransformandLoad(ETL)toolPentahoReportServer:ThisisagreatreportingenginepgAdmin3:Thisisanawesomedatabaseadministrationtoolphp5-pgsql:ThisisapackagethatallowsnativeaccesstoPostgreSQLfromPHPQCubed:ThisisthePHPdevelopmentframeworkwithPostgreSQLsupportYii:ThisisanothergreatPHPdevelopmentframeworkTalend:ThisisanotherETLtoolthatworks,butthisisnotmyfavoriteBIRT:ThisisagreatJAVAreportingtoolwithaneasyreportcreationenvironmentpsycopg2:ThisisthePythonbindingsforPostgreSQL

ThesetoolshavemadethePostgreSQLdevelopmentexperienceabreeze,andthisisnowherenearacompletelist.WecanfillthisbookwithjustalistofapplicationsthatsupportPostgreSQL,andthankstoitsliberallicense,PostgreSQLisembeddedinmanycommercialapplicationsthatyouneverreallyknow.

TipLessonlearned

Don’tworrytoomuchabouthowmanytoolsareouttherefortheproduct.Theonesthatmatterareavailable.

PlatformcompatibilitySQLServerisaMicrosoftproduct.Assuch,itwas,andwillalwaysbe,aMicrosoftplatformtool.ItisaccessibletosomelimiteddegreeviaODBC,butitisnotaseriouschoiceforcross-platformdevelopment.

MySQLandPostgreSQLsupporteveryoperatingsystemcurrentlyavailabletoday.Thisability(orthelackoflimitation)isastrongargumentforlong-termstability.Ifanyparticularoperatingsystemisnolongeravailable,ornolongersupportsopensourcesoftware,itisfairlysimpletomovethedatabaseservertoanotherplatform.

TipLessonlearned

Inthecommercialoperatingsystemwars,justsayno.

Applicationdesign “Thethingthathathbeen,itisthatwhichshallbe;andthatwhichisdoneisthatwhichshallbedone:andthereisnonewthingunderthesun.”

—Ecclesiastes1:9(KJV)

“…oldthingsarepassedaway;behold,allthingsarebecomenew.”

—2Corinthians5:16-18(KJV)

Insoftwaredevelopment,wearealwaysrunningintothesituationwherewhatisoldisnewagain,anddeveloperswhoembraceaphilosophyswearbyitlikeareligion.Weswingbackandforthbetweenthinserversandthinclients,flatandhierarchicalstorage,desktopapplicationsandwebapplicationsand,mostappropriatelyforthischapter,betweenclientandserverprogramming.

Thereasonforthisswingbetweenprogrammingimplementationshasgotnothingtodowiththefeaturesthattheclientortheserveroffers.Developerexperienceisamuchmorelikelyinfluence,andthisinfluencecangoineitherdirection,dependingonwhatthedeveloperencounteredfirst.

Iencourageboththeserver-centricdeveloperandtheclient-centricdevelopertolaydowntheirpitchforkswhilereadingtherestofthischapter.

Wewilldiscuss,induetime,mostofthenewfeaturesofserverprogramming.Ifyou’restillnotconvinced,wewilltakealookathowyoucanharnessthebenefitsofmostofthosefeatureswithoutleavingyourapplication-centeredpointofview.

DatabasesareconsideredharmfulThesimplestandleastpowerfulwayoflookingatserverprogramming,istoviewthedatabaseasadatabucket.UsingonlythemostbasicSQLstatementssuchasINSERT,SELECT,UPDATE,andDELETE,youcanmanipulatedata,asinglerowatatime,andcreateapplicationlibrariesformultipledatabaseseasily.

Thisapproachhassomemajordrawbacks.Movingdatabackandforthtothedatabaseserveronerowatatimeisextremelyinefficient,andyouwillfindthatthismethodissimplynotviableinaweb-scaleapplication.

Thisideaisusuallyassociatedwiththeconceptofadatabaseabstractionlayer,aclientlibrarythatallowsthedevelopertoswitchthedatabaseoutfromundertheapplicationwithlittleeffort.Thisabstractionlayerisveryusefulintheopensourcedevelopmentcommunity,whichallowstheuseofmanydatabases,buttheyhavenofinancialincentivetogetthebestpossibleperformance.

SQL,beingbasedonrelationalalgebraandtuplerelationalcalculus,hastheabilitytoquicklyandefficientlyperformset-basedprocessingonlargeamountsofdata;theapplication-sideprocessingusuallyinvolvesiterativelooping,whichisgenerallymuchslower.

Inmy27-yearcareer,Ihaveneveractuallychangedthedatabaseofaninstalled

applicationwithoutthrowingawaytheapplication.OneoftheprinciplesofagilesoftwaredevelopmentisYAGNI(youain’tgonnaneedit).Thisisoneofthosecases.

TipLessonlearned

Dataabstractionisvaluableforprojectsthatneedtoselectadatabaseplatformatinstallationtime.Foranythingelse,justsayno.

EncapsulationAnothertechniqueusedinmoreclient-centricdevelopmentphilosophies,istoisolatethedatabase-specificcallsintoalibraryofprocedures.Thisdesignisusuallyaimedatleavingtheapplicationincontrolofallthebusinesslogic.Theapplicationisstilltheking,andthedatabaseisstilljustanecessaryevil.

Thisviewofdatabasearchitecturesellstheapplicationdevelopershortbyignoringatoolboxfulloftoolsandchoosingonlythehammer.Everythingintheapplicationisthenpaintedtolooklikeanailandissmackedwiththehammer.

TipLessonlearned

Don’tgiveuponthepowerofthedatabasejustbecauseitisnotfamiliar.Useprocedurallanguagesandcheckoutextensiontoolkits.Therearesomeawesomepiecesofworkinthere.

WhatdoesPostgreSQLoffer?Sofar,we’vementionedprocedurallanguages,functions,triggers,customdatatypes,andoperators.ThesethingscanbecreateddirectlyinthedatabaseviatheCREATEcommandsoraddedaslibrariesusingextensions.

Now,wewillshowyousomethingsthatyouneedtokeepinmindwhenprogrammingontheserverinPostgreSQL.

DatalocalityIfpossible,keepthedataontheserver.Believeme,it’shappierthere,andperformanceismuchbetterwhenmodifyingdata.Ifeverythingwasdoneintheapplicationlayer,thedatawillneedtobereturnedfromthedatabasewiththemodificationsandthenfinallysentbacktothedatabaseforacommit.Ifyouarebuildingaweb-scalableapplication,thisshouldbeyourlastresort.

Let’swalkthroughasmallsnippetthatusestwomethodsinordertomakeanupdatetoasinglerecord:

<?php

$db=pg_connect("hostportuserpassworddbnameschema");

$sql="SELECT*FROMcustomerWHEREid=23";

$row=pg_fetch_array($db,$sql);

if($row['account_balance']>6000){

$sql="UPDATEcustomerSETvalued_customer=trueWHEREid=23;";;";

pg_query($db,$sql);

}

pg_close($db);

?>

Thiscodesnippetpullsarowofdatafromthedatabaseservertotheclient,makesanevaluation,andchangesacustomeraccountbasedontheevaluation.Theresultofthechangeisthensentbacktotheserverforprocessing.

Thereareseveralthingsthatarewrongwiththisscenario.First,thescalabilityisterrible.Imagineifthisoperationneededtobeperformedforthousands,orevenmillionsofcustomers.Thiswillbereallyslowbecausethecodewillprocesstherecordsonebyone,andeachrecordwillbesentoverthenetworkandthenupdated,whichinvolvesgoingoverthenetworkagainforeachrecord.

Thesecondproblemistransactionalintegrity.Whathappensiftheuser’saccountbalancechangesfromsomeothertransactionbetweenthequeryandtheupdate?Isthecustomerstillvalued?Thiswilldependonthebusinessreasonfortheevaluation.

Tryoutthefollowingexample:

<?php

$db=pg_connect('...');

pg_query('UPDATEcustomerSETvalued_customer=trueWHEREbalance>

6000;',$db);

pg_close($db);

?>

Thisexampleissimple,hastransactionalintegrity,andworksforanincrediblylargenumberofcustomers.Whypointoutsuchasimpleandobviousexample?Theanswerisbecausemanydevelopmentframeworksworkincorrectlybydefault.Thecodegeneratorwillproducesomeequivalentformofthisexampleintheinterestofbeingcross-platform,predictable,andeasytointegrateintoasimpledesignmodel.

Thismethodpromotesterriblepractices.Forsystemsthathaveaverylownumberofconcurrenttransactions,youwillprobablyseewhatyouexpect,butasconcurrencyincreases,thenumberofunintendedbehaviorsalsoincrease.

Thesecondexampleexposesabetterphilosophy:operateoncolumns(notonrows),leavethedataontheserver,andletthedatabasedothetransactionalworkforyou.That’swhatthedatabaseismadefor.

MorebasicsIthelpstohavesomebasicbackgroundinformationbeforeyoustartprogrammingfortheserver.Inthenextfewsections,wewillexplorethegeneraltechnicalenvironmentinwhichyouwillbeworking.Wewillcoveralotofinformation,butdon’tworrytoomuchaboutrememberingitallrightnow.Justtrytopickupthegeneralidea.

TransactionsThedefaulttransactionisolationlevelinPostgreSQLiscalledReadCommitted.Thismeansthatifmultipletransactionsattempttomodifythesamedata,theymustwaitforeachothertofinishbeforeactingontheresultingdata.Theywaitinafirst-come-first-serveorder.Thefinalresultofthedataiswhatmostpeoplewillnaturallyexpect:thelastchronologicalchangebeingreflected.

PostgreSQLdoesnotprovideanywaytodoadirtyread.Adirtyreadistheabilitytoviewthedatathewayitappearsinsomeoneelse’stransactionandtouseitasifitwerecommitted.ThisabilityisnotavailableinPostgreSQLbecauseofthewayinwhichthemultiversionconcurrencycontrolworks.

Thereareothertransactionisolationmethodsavailable;youcanreadaboutthemindetailathttp://www.postgresql.org/docs/current/static/transaction-iso.html.

Itisimportanttonote,thatwhennotransactionblocksarespecified(BEGIN..END),PostgreSQLwilltreateachindividualstatementlikeaprivatetransactionandcommitimmediatelywhenthestatementisfinished.Thisgivesothertransactionsachancetosettlebetweenyourstatements.Someprogramminglanguagesprovideatransactionblockaroundyourstatements,whilesomedonot.Pleasecheckyourlanguagedocumentationtofindoutwhetheryouarerunninginatransactedsession.

NoteWhenusingthetwomainclientstointeractwithPostgreSQL,thetransactionbehaviorisdifferent.Thepsqlcommand-lineclientdoesnotprovidetransactionblocks.Youareexpectedtoknowwhentostart/stopatransactiononyourown.ThepgAdmin3querywindow,ontheotherhand,wrapsanystatementthatyousubmitintoatransactionblockforyou.Thiswayitprovidesacanceloption.Ifthetransactionisinterrupted,ROLLBACKwillbeperformedandthedatabasewillgobacktoitsformerstate.

Someoperationsareexemptfromtransactions.Forexample,asequenceobjectwillcontinuetoincrementevenifthetransactionfailsandisrolledback.CREATEINDEXCONCURRENTLYrequiresthemanagementofitsowntransactionsandshouldnotbecalledfromwithinatransactionblock.ThesameistrueforVACUUM,aswellasCLUSTER.

GeneralerrorreportinganderrorhandlingIfyouwanttoprovideastatustotheuserduringexecution,youshouldbefamiliarwiththecommandsRAISE,NOTICE,andNOTIFY.Fromatransactionalperspective,thedifferenceisthatRAISEandNOTICEwillsendthemessageimmediately,evenwhenwrappedinatransaction,whileNOTIFYwillwaitforthetransactiontosettlebefore

http://www.postgresql.org/docs/current/static/transaction-iso.html

sendingamessage.NOTIFYwill,therefore,actuallynotnotifyyouofanythingifthetransactionfailsandisrolledback.

User-definedfunctionsTheabilitytowriteuser-definedfunctionsisthepowerhousefeatureofPostgreSQL.Functionscanbewritteninmanydifferentprogramminglanguages,canuseanykindofcontrolstructuresthatthelanguageprovides,andinthecaseof“untrusted”languages,canperformanyoperationthatisavailableinPostgreSQL.

FunctionscanprovidefeaturesthatarenotevendirectlyrelatedtoSQL.Someoftheupcomingexampleswillshowyouhowtogetnetworkaddressinformation,querythesystem,movefilesaround,anddojustaboutanythingthatyourheartdesires.

So,howdoweaccessthesugarygoodnessofPostgreSQL?Westartbydeclaringthatwewantafunction:

CREATEORREPLACEFUNCTIONaddition(integer,integer)RETURNSinteger

AS$$

DECLAREretvalinteger;

BEGIN

SELECT$1+$2INTOretval;

RETURNretval;

END;

$$LANGUAGEplpgsql;

Whatifwewanttoaddthreeintegerstogether?Usingthefollowingcodewecanaddthreeintegerstogether:

CREATEORREPLACEFUNCTIONaddition(integer,integer,integer)RETURNS

integer

AS$$


BEGIN

SELECT$1+$2+$3INTOretval;

RETURNretval;

END;

$$LANGUAGEplpgsql;

Wejustinvokedaconceptcalledfunctionoverloading.Thisfeatureallowsyoutodeclareafunctionofthesamename,butwithdifferentparametersthatpotentiallybehavedifferently.Thisdifferencecanbejustassubtleaschangingthedatatypeofoneoftheargumentstothefunction.ThefunctionthatPostgreSQLinvokesdependsontheclosestmatchtothefunctionargumentsandexpectedreturntype.

Supposewewanttoaddtogetheranynumberofintegers?Well,PostgreSQLhasawaytodothisalso,asfollows:

CREATEORREPLACEFUNCTIONaddition(VARIADICarrinteger[])RETURNS

integer

AS$$


BEGIN

SELECTsum($1[i])INTOretvalFROMgenerate_subscripts($1,1)g(i);

RETURNretval;

END;

$$

LANGUAGEplpgsql;

Thiswillallowyoutopassinanynumberofintegersandgetanappropriateresponse.Thesefunctions,ofcourse,donothandlerealornumericdatatypes.Tohandleotherdatatypes,simplydeclarethefunctionagainwiththosetypesandcallthemwiththeappropriateparameters.

Formoreinformationaboutvariableparameters,checkouthttp://www.postgresql.org/docs/9.3/static/xfunc-sql.html#XFUNC-SQL-VARIADIC-FUNCTIONS.

OtherparametersThereismorethanonewaytogetdataintoafunctionandoutofit.WecanalsodeclareIN/OUTparameters,returnatable,returnasetofrecords,andusecursorsforboththeinputandoutput.

ThisbringsustoapseudotypecalledANY.Itallowstheparametertypetobeundefined,anditallowsanybasicdatatypetobepassedtothefunction.Then,itisuptothefunctiontodecidewhattodowiththedata.ThereareseveralotherpseudotypesavailableinPostgreSQL,alsocalledthepolymorphictypes.Theseareanyelement,anyarray,anynonarray,anyenum,andanyrange.Youcanreadmoreaboutthesepseudotypesathttp://www.postgresql.org/docs/9.3/static/datatype-pseudo.html.

Hereisanexampleofthepseudotypeanyarray.Thefollowingsimplefunctiontakesanarrayofanytypeasanargumentandreturnsanarraybyremovingalltheduplicates:

CREATEORREPLACEFUNCTIONremove_duplicates(anyarray)

RETURNSanyarrayAS

$$

SELECTARRAY(SELECTDISTINCTunnest($1));

$$

LANGUAGE'sql';

postgres=#SELECTremove_duplicates(ARRAY[1,1,2,2,3,3]);

remove_duplicates

-------------------

{1,2,3}

(1row)

postgres=#SELECTremove_duplicates(ARRAY['a','a','b','c']);

remove_duplicates

-------------------

{b,a,c}

(1row)

MorecontrolOnceyouhaveyourfunctionwrittenthewayyouneed,PostgreSQLgivesyouadditionalcontroloverhowthefunctionexecutes.Youcancontrolwhatdatathefunctioncanaccess

http://www.postgresql.org/docs/9.3/static/xfunc-sql.html#XFUNC-SQL-VARIADIC-FUNCTIONS

http://www.postgresql.org/docs/9.3/static/datatype-pseudo.html

andhowPostgreSQLwillinterprettheexpenseofrunningthefunction.

Therearetwostatementsthatprovideasecuritycontextforyourfunctions.ThefirstoneisSECURITYINVOKER,whichisthedefaultsecuritycontext.Inthedefaultcontext,theprivilegesofthecallinguserarerespectedbythefunction.

TheothercontextisSECURITYDEFINER.Inthiscontext,theuserprivilegesofthecreatorofthefunctionarerespectedduringtheexecutionofthefunction.Generally,thisisusedtotemporarilyescalateuserrightsforaspecificpurpose.

Thisisveryusefulifyouwanttohaveastrictercontroloveryourdata.Inanenvironmentwheresecurityoftheunderlyingdataisimportant,andyoudon’twantuserstodirectlySELECTthedataorchangeitusingINSERT,UPDATE,orDELETE,youcancreateasetoffunctionswiththesecurity-definerattributeasAPIsforthetables.ThisapproachgivesyoucompletecontroloveryourAPIbehaviorandhowuserscanaccesstheunderlyingobjects.

Costcanalsobedefinedforthefunction.Thiscostwillhelpthequeryplannerestimatehowexpensiveitistocallthefunction.Higherordersofcostwillcausethequeryplannertochangetheaccesspath,soyourfunctionwillbecalledasfewtimesaspossible.ThePostgreSQLdocumentationshowsthesenumberstobeafactorofcpu_operator_cost.That’smorethanalittlemisleading.ThesenumbershavenodirectcorrelationtoCPUcycles.Theyareonlyrelevantincomparisonwithoneanother.It’smorelikehowsomenationalmoneycompareswiththerestoftheEuropeanUnion.SomeEurosaremoreequalthanothers.

Toestimateyourownfunction’scomplexity,startwiththelanguageyouareimplementingitin.ForC,thedefaultwillbe1*numberofrecordsreturned,anditwillbe100forallotherlanguages,accordingtothePostgreSQLdocumentationathttp://www.postgresql.org/docs/current/static/sql-createfunction.html.Forplsh,youmaywanttouse150ormore,dependingonhowmanyexternaltoolcallsareinvolvedingettingananswer.Thedefaultis100andthisseemstoworkreasonablywellforPL/pgSQL.

http://www.postgresql.org/docs/current/static/sql-createfunction.html

SummaryNowyouknowafewthingsaboutthePostgreSQLenvironment,aswellassomethingsthatwillhelpyouintheunforeseeablefuture.PostgreSQLisbuilttohandleyourneeds,butmoreimportantly,itisbuiltnottochangeunderneathyouinthefuture.

WetouchedupontheenvironmentandcalledoutsomeofthemoreimportantthingstobekeptinmindwhenprogrammingontheserverinPostgreSQL.Don’tworrytoomuchifyoudon’trememberallofit.Itisfinetogoontothenextchapter,wherewewillactuallystartmakingsomeusefulfunctionsandlearnaboutwritingourfirstPL/pgSQLfunctions.Youwillalsolearnhowtowriteconditionalstatements,loops,anddifferentwaystoreturndata.Then,comebackandreviewthischapterwhenyouhaveaclearerunderstandingofthefeaturesavailabletothefunctionwriter.

Chapter3.YourFirstPL/pgSQLFunctionAfunctionisthebasicbuildingblockforextendingPostgreSQL.Afunctionacceptsinputintheformofparameters,anditcancreateoutputintheformofoutputparametersorreturnvalues.ManyfunctionsareprovidedbyPostgreSQLitself,thatis,commonmathematicalfunctionssuchassquarerootsandabsolutevalues.Foracomprehensivelistofthefunctionsthatarealreadyavailable,gotohttp://www.postgresql.org/docs/current/static/functions.html.

Thefunctionsthatyoucreatehavethesameprivilegesandabilitythatthebuilt-infunctionspossess.ThedevelopersofPostgreSQLusethesamelibrariestoextendthedatabasethatyouuse,asadeveloper,towriteyourbusinesslogic.

Thismeans,thatyouhavethetoolsavailabletobeafirst-classcitizenofthePostgreSQLdevelopmentcommunity.Infact,therearenosecond-classseatsonthisbus.

AfunctionacceptsparametersthatcanbeofanydatatypeavailableinPostgreSQL,anditreturnsresultstothecallerusingthesametype.Whatyoudowithinthefunctionisentirelyuptoyou.YouhavebeenempoweredtodoanythingthatPostgreSQLiscapableofdoing.YouareherewithalsowarnedthatyouarecapableofdoinganythingthatPostgreSQLiscapableofdoing.Thetrainingwheelsareoff.

Inthischapter,youwilllearnthefollowingtopics:

ThebasicbuildingblocksofaPostgreSQLfunctionPassingparametersintoafunctionThebasiccontrolstructuresinsideafunctionReturningresultsoutofafunction

http://www.postgresql.org/docs/current/static/functions.html

WhyPL/pgSQL?PL/pgSQLisapowerfulSQLscriptinglanguage,thatisheavilyinfluencedbyPL/SQL,thestoredprocedurelanguagedistributedwithOracle.ItisincludedinthevastmajorityofPostgreSQLinstallationsasastandardpartoftheproduct,soitusuallyrequiresnosetupatalltobegin.

PL/pgSQLalsohasadirtylittlesecret.ThePostgreSQLdevelopersdon’twantyoutoknowthatitisafull-fledgedSQLdevelopmentlanguage,capableofdoingprettymuchanythingwithinthePostgreSQLdatabase.

Whyisthisasecret?Foryears,PostgreSQLdidnotclaimtohavestoredprocedures.PL/pgSQLfunctionswereoriginallydesignedtoreturnscalarvaluesandwereintendedforsimplemathematicaltasksandtrivialstringmanipulations.

Overtheyears,PL/pgSQLdevelopedarichsetofcontrolstructuresandgainedtheabilitytobeusedbytriggers,operators,andindexes.Intheend,developersweregrudginglyforcedtoadmitthattheyhadacomplete,storedproceduredevelopmentsystemontheirhands.

Alongtheway,thegoalofPL/pgSQLchangedfromsimplescalarfunctions,toprovidingaccesstoallofthePostgreSQLsysteminternals,withfullcontrolstructure.Thefulllistofwhatisavailableinthecurrentversionisprovidedathttp://www.postgresql.org/docs/current/static/plpgsql-overview.html.

Today,thefollowingaresomeofthebenefitsofusingPL/pgSQL:

ItiseasytouseItisavailablebydefaultonmostdeploymentsofPostgreSQLItisoptimizedfortheperformanceofdata-intensivetasks

InadditiontoPL/pgSQL,PostgreSQLalsoallowsmanyotherlanguagessuchasPL/Perl,PL/Python,PL/Proxy,andPL/Tcltobepluggedintothedatabase,someofwhichwillbecoveredinthisbook.YoumayalsochoosetowriteyourfunctionsinPerl,Python,PHP,bash,andahostofotherlanguages,buttheywilllikelyneedtobeaddedtoyourinstanceofPostgreSQL.

http://www.postgresql.org/docs/current/static/plpgsql-overview.html

ThestructureofaPL/pgSQLfunctionItdoesn’ttakemuchtogetaPL/pgSQLfunctionworking.Here’sabasicexample:

CREATEFUNCTIONmid(varchar,integer,integer)RETURNSvarchar

AS$$

BEGIN

RETURNsubstring($1,$2,$3);

END;

$$

LANGUAGEplpgsql;

TheprecedingfunctionshowsthebasicelementsofaPL/pgSQLfunction.Itcreatesanaliasforthesubstringbuilt-infunctioncalledmid.ThisisahandyaliastohavearoundfordevelopersthatcomefromMicrosoftSQLServerorMySQLandarewonderingwhathappenedtothemidfunction.Italsoillustratesthemostbasicparameter-passingstrategy:parametersarenotnamedandareaccessedinthefunctionbytheirrelativelocationfromlefttoright.The$$characterinthisexamplerepresentsthestartandendofthecodeblock.Thischaractersequencecanbearbitraryandyoucanusesomethingelseofyourchoice,butthisbookuses$$inalltheexamples.

ThebasicelementsofaPL/pgSQLfunctionarename,parameters,returntype,body,andlanguage.Itcanbearguedthatparametersarenotmandatoryforafunctionandneitheristhereturnvalue.Thismightbeusefulforaprocedurethatoperatesondatawithoutprovidingaresponse,butitwillbeprudenttoreturnthevalueTRUEtoindicatethattheproceduresucceeded.

AccessingfunctionargumentsFunctionargumentscanalsobepassedandaccessedbyname,insteadofjustbytheordinalorder.Byaccessingtheparametersbyname,itmakestheresultingfunctioncodealittlemorereadable.Thefollowingisanexampleofafunctionthatusesnamedparameters:

CREATEFUNCTIONmid(keyfieldvarchar,starting_pointinteger)RETURNS

varchar

AS

$$

BEGIN

RETURNsubstring(keyfield,starting_point);

END

$$

LANGUAGEplpgsql;

Theprecedingfunctionalsodemonstratestheoverloadingofthemidfunction.OverloadingisanotherfeatureofPostgreSQLfunctions,whichallowsmultipleprocedurestousethesamename,butadifferentnumberortypesofparameters.Inthiscase,wefirstdeclaredthemidfunctionwiththreeparameters.However,inthisexample,overloadingisusedtoimplementanalternativeformofthemidfunction,wherethereareonlytwoparameters.Whenthethirdparameterisomitted,theresultwillbeastringstartingfromstarting_pointandcontinuingtotheendoftheinputstring,asshownhere:

SELECTmid('KirkL.Roybal',9);

Theprecedinglineofcodeyieldsthefollowingresult:

mid

--------

Roybal

(1row)

Inordertoaccessthefunctionparametersbyname,PostgreSQLmakesafeweducatedguessesdependingonthestatement.Consider,foramoment,thefollowingfunction:

CREATEORREPLACEFUNCTIONambiguous(parametervarchar)RETURNSintegerAS

$$


BEGIN

INSERTINTOparameter(parameter)VALUES(parameter)RETURNINGidINTO

retval;

RETURNretval;

END;

$$

LANGUAGEplpgsql;

SELECTambiguous('parameter');

Thisisanexampleofpositivelyatrociousprogrammingsincetheargument,table,andits

columnareallcalledparameter.Thisshouldneveroccuroutsideanexampleofhownottowritefunctions.However,PostgreSQLisintelligentenoughtocorrectlydeducethatthecontentsofthefunctionparameterareonlylegalintheVALUESlist.AllotheroccurrencesofparameterareactuallyphysicalPostgreSQLentities.

Wealsointroducedanoptionalsectiontothefunction.WedeclareavariablebeforetheBEGINstatement.Variablesthatappearinthissectionarevalidduringtheexecutionofthefunction.

Alsonote,theRETURNINGidINTOretvalstatementinthisfunction.Thisfeatureallowsthedevelopertospecifytheidentityfieldoftherecordandreturnsthevalueofthisfieldaftertherecordhasbeeninserted.Ourfunctionthenreturnsthisvaluetothecallerasanindicatorthatthefunctionsucceeded,andasawaytofindtherecordthathasbeeninserted.Thisisagoodwaytoreturnvaluesinsertedbydefault,suchastheserialsequencenumbers.Youcanuseanyexpressionwithtablecolumnnames,andthesyntaxwillbesimilartothecolumnlistinaSELECTstatement.

ConditionalexpressionsConditionalexpressionsallowdeveloperstocontroltheactionofthefunction,basedonadefinedcriteria.PostgreSQLprovidestheCASEandIFstatementstoexecutedifferentcommandsbasedonconditions.ThefollowingisanexampleoftheusageofaCASEstatementtocontrolhowastringistreatedbasedonitsvalue.Ifthevalueisnullorcontainsazero-lengthstring,itistreatedthesameasnull:

CREATEORREPLACEFUNCTIONformat_us_full_name(

prefixtext,firstnametext,

mitext,lastnametext,

suffixtext)

RETURNStextAS

$$

DECLARE

fname_mitext;

fmi_lnametext;

prefix_fmiltext;

pfmil_suffixtext;

BEGIN

fname_mi:=CONCAT_WS('',

CASEtrim(firstname)

WHEN''

THENNULL

ELSEfirstname

END,

CASEtrim(mi)

WHEN''

THENNULL

ELSEmi

END||'.');

fmi_lname:=CONCAT_WS('',

CASEfname_mi

WHEN''

THENNULL

ELSEfname_mi

END,

CASEtrim(lastname)

WHEN''

THENNULL

ELSElastname

END);

prefix_fmil:=CONCAT_WS('.',

CASEtrim(prefix)

WHEN''

THENNULL

ELSEprefix

END,

CASEfmi_lname

WHEN''

THENNULL

ELSEfmi_lname

END);

pfmil_suffix:=CONCAT_WS(',',

CASEprefix_fmil

WHEN''

THENNULL

ELSEprefix_fmil

END,

CASEtrim(suffix)

WHEN''

THENNULL

ELSEsuffix||'.'

END);

RETURNpfmil_suffix;

END;

$$

LANGUAGEplpgsql;

Theideahere,isthatwhenanyelementofafullnameismissing,thesurroundingpunctuationandwhitespacesshouldalsobemissing.Thisfunctionreturnsawell-formattedfullnameofapersonintheUSA,withasmuchofthenamefilledinaspossible.Whenrunningthisfunction,youwillseethefollowing:

postgres=#SELECTformat_us_full_name('Mr','Martin','L','King','Jr');

format_us_full_name

-------------------------

Mr.MartinL.King,Jr.

(1row)

Now,let’strywithanamemissing:

postgres=#SELECTformat_us_full_name('','Martin','L','King','Jr');

format_us_full_name

---------------------

MartinL.King,Jr.

(1row)

Anotherwaytouseconditionalexpressions,isusingtheIF/THEN/ELSEblocks.Thefollowingisthesamefunctionagain,writtenusingIFstatements,ratherthanCASEstatements:

CREATEORREPLACEFUNCTIONformat_us_full_name(

prefixtext,firstnametext,

mitext,lastnametext,

suffixtext)

RETURNStextAS

$$

DECLARE

fname_mitext;

fmi_lnametext;

prefix_fmiltext;

pfmil_suffixtext;

BEGIN

fname_mi:=CONCAT_WS('',

IF(trim(firstname)='',NULL,firstname),

IF(trim(mi)='',NULL,mi||'.')

);

fmi_lname:=CONCAT_WS('',

IF(fname_mi='',NULL,fname_mi),

IF(trim(lastname)='',NULL,lastname)

);

prefix_fmil:=CONCAT_WS('.',

IF(trim(prefix)='',NULL,prefix),

IF(fmi_lname='',NULL,fmi_lname)

);

pfmil_suffix:=CONCAT_WS(',',

IF(prefix_fmil='',NULL,prefix_fmil),

IF(trim(suffix)='',NULL,suffix||'.')

);

RETURNpfmil_suffix;

END;

$$

LANGUAGEplpgsql;

PostgreSQL’sPL/pgSQLprovidesseveralothersyntacticalvariantsoftheseconditionalexpressions.Thisintroductionfocusesonthemostcommonlyusedones.Foramorecompletediscussionofthetopic,visithttp://www.postgresql.org/docs/current/static/functions-conditional.html.

http://www.postgresql.org/docs/current/static/functions-conditional.html

LoopswithcountersThePL/pgSQLlanguageprovidesasimplewaytoloopthroughsomeelements.ThefollowingisafunctionthatreturnsthenthFibonaccisequencenumber:

CREATEORREPLACEFUNCTIONfib(ninteger)

RETURNSINTEGERAS$$

DECLARE

counterinteger:=0;

ainteger:=0;

binteger:=1;

BEGIN

IF(n<1)THEN

RETURN0;

ENDIF;

LOOP

EXITWHENcounter=n;

counter:=counter+1;

SELECTb,a+bINTOa,b;

ENDLOOP;

RETURNa;

END;

$$

LANGUAGEplpgsql;

SELECTfib(4);

Theprecedingcodegives3astheoutput.

Justfortherecord,eachelementintheFibonaccisequenceisthesumoftheprevioustwoelements.Thus,thefirstfewelementsofthesequenceshouldbe0,1,1,2,3,5,8,13,21,34,andsoon.ThereareafewPostgreSQLFibonaccisequencefunctionsoutthereontheinterwebs,buttheyusethedreadedrecursivemethod.Inthiscase,recursionisabadthing.Ourexampleusestheiterativemethodthatavoidstheuseofstacksforrecursion,asthismightresultinthelackofstackspaceforlargenumbers.

Inthisfunction,wealsointroduceddefaultvaluestothevariablesinthedeclarationssection.Whenthefunctionisinvoked,thevariableswillbeinitiallysettothesevalues.

Also,takeaquickganderatthestatementSELECTb,a+bINTOa,b.Thisstatementmakestwovariableassignmentsatthesametime.Itavoidstheuseofathirdvariablewhileactingonbothaandb.

AnotherslightvariationofthefunctionusingaFORloop,isasfollows:

CREATEORREPLACEFUNCTIONfib(ninteger)

RETURNSINTEGER

AS$$

DECLARE

counterinteger:=0;

ainteger:=0;

binteger:=1;

BEGIN

IF(n<1)THEN

RETURN0;

ENDIF;

FORcounterIN1..n

LOOP

SELECTb,a+bINTOa,b;

ENDLOOP;

RETURNa;

END;

$$

LANGUAGEplpgsql;

Forsomeadditionalloopingsyntax,takealookatthePostgreSQLdocumentationpageathttp://www.postgresql.org/docs/current/static/plpgsql-control-structures.html.

StatementterminationInPL/pgSQL,allblocksandthestatementswithintheblocks,mustendwithasemicolon.TheexceptionsarethestatementsthatstartablockwithIForBEGIN.Block-startingstatementsarenotcompletestatements;therefore,thesemicolonisalwaysaftertheblock-endingstatement,suchasEND;orENDIF;.

http://www.postgresql.org/docs/current/static/plpgsql-control-structures.html

LoopingthroughqueryresultsBeforeweembarkonthisjourneyofqueryresultloops,IthinkIshouldwarnyou,thatifyouareusingthismethod,youareprobablydoingitwrong.Thisisoneofthemostprocessor-andmemory-intensiveoperationsthatPostgreSQLoffers.Thereareexceedinglyfewreasonstoiteratethrougharesultsetonthedatabaseserverthatoffsetthiscost.IwouldencourageyoutothinkhardabouthowtoimplementthesameideausingaVALUESlistinaquery,temporarytable,andpermanenttable,ortoprecomputethevaluesinanywaypossible,inordertoavoidthisoperation.So,doyoustillthinkyouhaveanoverwhelmingreasontousethistechnique?Okay,thenreadon.Thefollowingisthesimpleversion:

FORrowIN

EXECUTE'SELECT*FROMjob_queueqWHERENOTprocessedLIMIT100'

LOOP

CASErow.process_type

WHEN'archive_point_of_sale'

THENINSERTINTOhist_orders(...)

SELECT…FROMorders

INNERJOINorder_detail…

INNERJOINitem…;

WHEN'prune_archived_orders'

THENDELETEFROMorder_detail

WHEREorder_idin(SELECTorder_idFROMhist_orders);

DELETEFROMorders

WHEREorder_idIN(SELECTorder_idFROMhist_orders);

ELSE

RAISENOTICE'Unknownprocess_type:%',row.process_type;

END;

UPDATEjob_queueSETprocessed=TRUEWHEREid=q.id;

ENDLOOP;

Theprecedingexampleshowsabasicstrategypatternofprocessingmessagesinajobqueue.Usingthistechnique,therowsinatablecontainalistofjobsthatneedtobeprocessed.

WeintroducetheEXECUTEstatementhereaswell.TheSELECTstatementisastringvalue.UsingEXECUTE,wecandynamicallybuildPL/pgSQLcommandsasstringsandtheninvokethemasstatementsagainstthedatabase.Thistechniquecomesinhandy,whenwewanttochangethetablenameorotherSQLkeywordsthatmakeupourstatement.ThesepartsoftheSQLstatementcannotbestoredinvariablesandarenotgenerallychangeable.WithEXECUTE,wecanchangeanypartofthestatementwejollywellplease.WemustmentionthatEXECUTEhasacostassociatedwithit:thequeriesarepreparedeachtimebeforerunning.

ThefollowingisanexamplefromthePostgreSQLdocumentationthatshowsdynamiccommandsrunninginsidealoop:

CREATEFUNCTIONcs_refresh_mviews()RETURNSintegerAS$$

DECLARE

mviewsRECORD;

BEGIN

PERFORMcs_log('Refreshingmaterializedviews…');

FORmviewsINSELECT*FROMcs_materialized_viewsORDERBYsort_key

LOOP

—Now"mviews"hasonerecordfromcs_materialized_views

PERFORMcs_log('Refreshingmaterializedview'||

quote_ident(mviews.mv_name)||'...');

EXECUTE'TRUNCATETABLE'||quote_ident(mviews.mv_name);

EXECUTE'INSERTINTO'||quote_ident(mviews.mv_name)||''||

mviews.mv_query;

ENDLOOP;

PERFORMcs_log('Donerefreshingmaterializedviews.');

RETURN1;

END;

$$LANGUAGEplpgsql;

Theprecedingloopingexample,showsamorecomplexfunctionthatrefreshesthedatainsomestagingtables.Thesestagingtablesaredesignatedmaterializedviewsbecausethedataisactuallyphysicallytransferredtothestagingtables.Thismethodwasacommonwaytoreducequeryexecutionoverheadformanypresentationsofthesamedata,beforematerializedviewswereofficiallysupportedinPostgreSQL9.3.Inthiscase,theinefficiencyofloopingistrivialcomparedtothecontinuedcostofrepeatedqueriestothesamedata.

PERFORMversusSELECTYoumayhavenoticedastatementinthepreviousexamplethatwehaven’tcoveredyet.UsethePERFORMcommandwhenyouwanttojustdiscardtheresultsofastatement.Changethepreviousexampletothefollowingline:

SELECTcs_log("Donerefreshingmaterializedviews");

ThequeryenginewillreturnNodestinationforresultdata.

Wecanretrievetheresultsintovariables,andthenproceedtoignorethevariables,butthat’sjustalittletoosloppyformytaste.ByusingthePERFORMstatement,wehaveindicatedthatignoringtheresultswasnotaccidental.Wewerehappywiththefactthatthelogwasappendedtoblindly,andifitwasn’t,ohwell,wedidn’tfailtocontinuetheexecutionbecauseofalogentryissue.

LoopingThroughArraysThereisaveryconvenientloopconstructcalledFOREACH,whichallowsyoutoloopthroughtheelementsofanarray.Let’sdiveintoanexample:

CREATEFUNCTIONfindmax(int[])RETURNSint8AS$$

DECLARE

maxint8:=0;

xint;

BEGIN

FOREACHxINARRAY$1

LOOP

IFx>maxTHEN

max:=x;

ENDIF;

ENDLOOP;

RETURNmax;

END;

$$LANGUAGEplpgsql;

Thisfunctionisquiteself-explanatory.Itfindsthemaximumvaluesfromagivenintegerarray.ThepointtobenotedhereisthatunlikeanormalFORloop,aFOREACHloopcanonlyuseacountervariablethatisalreadydeclared.Youcanseethatwehavedeclaredxbeforeweuseditasacounterintheloop.Youcanrunthisfunctiontoseeifitworks:

postgres=#selectfindmax(ARRAY[1,2,3,4,5,-1]);

findmax

---------

5

(1row)

ReturningarecordSofar,allofourfunctionexampleshavefeaturedasimplescalarvalueintheRETURNclause.InPL/pgSQL,youcanalsodefineset-returningfunctions(SRF).Thesefunctionscanreturneitheratypedefinedbyanexistingtableoragenericrecordtype.Let’stakealookatasimpleexample:

CREATETABLEnames(idserial,namevarchar);

INSERTINTOnames(name)VALUES('John');

INSERTINTOnames(name)VALUES('Martin');

INSERTINTOnames(name)VALUES('Peter');

CREATEORREPLACEFUNCTIONGetNames()RETURNSSETOFnamesAS'SELECT*FROM

names;'LANGUAGE'sql';

Wejustdefinedaverysimplefunction,GetNames(),whichwillsimplyreturnalltherowsfromournewlydefinednamestable.

IfyouruntheGetNames()functionnow,youwillgetthefollowingoutput:

postgres=#selectGetNames();

getnames

------------

(1,John)

(2,Martin)

(3,Peter)

(3rows)

YoucanuseanSRFinplaceofatableorasasubqueryintheFROMclauseofaquery.Here’sanexample:

postgres=#select*fromGetNames()whereid>2;

id|name

----+-------

3|Peter

(1row)

Inadditiontothetabletypes,wecanalsoreturngenerictypesfromanSRF.Wecanchangeourexamplealittletodemonstratethis.Let’sdefineanewreturntypeandanewfunction:

CREATETYPEnametypeAS(idint,namevarchar);

CREATEFUNCTIONPlpgGetNames()RETURNSSETOFnametypeAS

$$

DECLARE

rnametype%rowtype;

BEGIN

FORrINSELECTid,nameFROMnamesLOOP

RETURNNEXTr;

ENDLOOP;

RETURN;

END;

$$

LANGUAGE'plpgsql';

ThePlpgGetNames()functiondeclaresavariablertobeofrowtypenametype.Thisvariableisusedtostoretherowsqueriedintheloop.Thefunctiondoesaloopoverthenamestableandsetsrtoeachrowintheresultset.TheRETURNNEXTcommandmeansthatanoutputrowisqueuedintothereturnsetofthefunction.Thisdoesnotcausethefunctiontoreturn.Finally,theRETURNstatementaftertheloopreturnsalltherows,whichwerequeuedearlierusingRETURNNEXT.

Let’srunournewfunction,asshownhere:

postgres=#SELECTPlpgGetNames();

plpggetnames

--------------

(1,John)

(2,Martin)

(3,Peter)

(3rows)

Forthesakeofasecondexample,wewillassumethatyouareinthemiddleofabigsoftwaredevelopmentupgradeprocedurethatusesaname/valuepairtablestructuretostoresettings.Youhavebeenaskedtochangethetablestructurefromthekeyandvaluecolumnstoaseriesofcolumns,wherethecolumnnameisnowthenameofthekey.ThisissimilartothepivottablesinExcel.Bytheway,youalsoneedtopreservethesettingsforeveryversionofthesoftwareyouhaveeverdeployed.

IfyoutakealookattheexistingCREATETABLEstatementforthetableyouhavetoworkwith,youwillfindthefollowing:

CREATETABLEapplication_settings_old(

versionvarchar(200),

keyvarchar(200),

valuevarchar(2000));

WhenyourunaSELECTstatementagainstthetable,youwillfindoutthattherearen’tmanysettings,buttherehavebeenquiteafewversionsofthesettings.So,let’smakeanewtablethatisalittlemoreexplicit:

CREATETABLEapplication_settings_new(

versionvarchar(200),

full_namevarchar(2000),

descriptionvarchar(2000),

print_certificatevarchar(2000),

show_advertisementsvarchar(2000),

show_splash_screenvarchar(2000));

TransformingthesettingsdataintothisnewformatcanbeaccomplishedwithanINSERTstatementandafunctionthatconvenientlyreturnsourdatainthenewtableformat.

Let’saddsometestdata,asfollows:

INSERTINTOapplication_settings_old

VALUES('3456','full_name','test_name');


VALUES('3456','description','test_description');


VALUES('3456','print_certificate','yes');


VALUES('3456','show_advertisements','yes');


VALUES('3456','show_splash_screen','no');

Let’sgoaheadanddefinethefunction:

CREATEORREPLACEFUNCTION

flatten_application_settings(app_versionvarchar(200))

RETURNSsetofapplication_settings_new

AS$$

BEGIN

—Createatemporarytabletoholdasinglerowofdata

IFEXISTS(SELECTrelnameFROMpg_classWHERErelname='tmp_settings')

THEN

TRUNCATETABLEtmp_settings;

ELSE

CREATETEMPTABLEtmp_settings(LIKEapplication_settings_new);

ENDIF;

--therowwillcontainallofthedataforthisversion

INSERTINTOtmp_settings(version)VALUES(app_version);

—addthedetailstotherecordforthisapplicationversion

UPDATEtmp_settings

SETfull_name=(SELECTvalue

FROMapplication_settings_old

WHEREversion=app_version

ANDkey='full_name'),

description=(SELECTvalue



ANDkey='description'),

print_certificate=(SELECTvalue



ANDkey='print_certificate'),

show_advertisements=(SELECTvalue



ANDkey='show_advertisements'),

show_splash_screen=(SELECTvalue



ANDkey='show_splash_screen');

--handbacktheresultstothecaller

RETURNQUERYSELECT*FROMtmp_settings;

END;

$$LANGUAGEplpgsql;

Theprecedingfunctionreturnsasinglerowofdatatothecallingquery.Therowcontainsallthesettingsthatwerepreviouslydefinedaskey/valuepairs,butareexplicitlydefinedfieldsnow.Thefunctionandthefinaltablecanalsobeenhancedtotransformthedatatypesofthesettingstosomethingmoreexplicit.However,I’llleavethatoneuptoyou.

Wethenproceedtousethefunctioninordertodothetransformation:

INSERTINTOapplication_settings_new

SELECT(flatten_application_settings(version)).*

FROM(

SELECTversion


GROUPBYversion)ASver;

Voilá!Thedataisnowavailableinatabularforminthenewtablestructure.

Actingonthefunction’sresultsThepreviousexampleshowedonewaytoretrieve,andfurtherprocess,functionresults.Thefollowingareafewmoreusefulwaystocallafunction:

SELECTfib(25);

SELECT(flatten_application_settings('9.08.97')).*;

SELECT*FROMflatten_application_settings('9.08.97');

AnyoftheprecedingmethodswillcreatealegalfieldlistinPostgreSQL,which,inturn,canbeusedinanywaythefieldsinasimpleSELECTstatementonatableareused.

Theexampleintheprevioussectionusedtheresultsoftheflatten_application_settings()function,asourceofdataforanINSERTstatement.ThefollowingisanexampleofhowtousethesamefunctionasadatasourceforUPDATE:

UPDATEapplication_settings_new

SETfull_name=flat.full_name,

description=flat.description,

print_certificate=flat.print_certificate,

show_advertisements=flat.show_advertisements,

show_splash_screen=flat.show_splash_screen

FROMflatten_application_settings('9.08.97')flat;

Usingtheapplicationversionasakey,wecanupdatetherecordsinthenewtable.Isn’tthisareallyhandywaytokeepupwiththechangestotheapplicationsettings,whileboththeoldandnewapplicationsarestillactive?I’lltakeanycomplimentsintheformofcash(orbeer),please.

SummaryWritingfunctionsinPostgreSQLisanextremelypowerfultool.PostgreSQLfunctionsprovidetheabilitytoaddfunctionalitytothedatabasecore,inordertoincreaseperformance,security,andmaintainability.

Thesefunctionscanbewritteninjustaboutanylanguagethatisavailabletotheopensourcecommunityandseveralthatareproprietary.Ifthelanguagethatyouwanttowritetheminisnotavailable,itcanbemadeavailablequicklyandeasilythroughaveryrobustandcompletecompatibilitylayer.Inthischapter,weonlylookedatPL/pgSQLfunctions.

Inthenextchapter,wewillfocusmoreonPL/pgSQLfunctions,andtakealookatthedifferentwaysinwhichdatacanbereturnedusingOUTparametersandreturnvalues.

Chapter4.ReturningStructuredDataInthepreviouschapter,wesawfunctionsthatreturnsinglevalues.Thesefunctionsreturneithera“scalar,”simpletypesuchasaninteger,text,ordata;oramorecomplextype,similartoarowinthedatabasetable.Inthischapter,wewillexpandtheseconceptsandshowyouhowtoreturnyourdatatotheclientinmorepowerfulways.

Wewillalsoexaminethefollowingtopics:

DifferencesbetweenSETOFscalars,rows,andarraysReturningCURSORs,whicharekindof“lazy”tables,thatis,somethingthatcanbeusedtogetasetofrows,butwhichmaynothaveactuallyevaluatedorfetchedtherowsyet,asthemodernworldisnotaboutrigidtable-structureddataWaystodealwithmorecomplexdatastructures,bothpredefinedanddynamicallycreated

Let’sstartwithasimpleexampleandthenaddmorefeaturesandvariantsaswego.

SetsandarraysRowsetsaresimilartoarraysinmanyways,buttheymainlydifferintheirusage.Formostdatamanipulations,youwillwanttouserowsets,astheSQLlanguageisdesignedtodealwiththem.Arrays,however,aremostusefulforstaticstorage.Theyaremorecomplicatedforclientapplicationstousethanrowsets,withusabilityfeaturesmissing,suchasnosimpleandstraightforwardbuilt-inwaystoiterateoverthem.

ReturningsetsWhenyouwriteasetreturningfunction,therearesomedifferencesfromanormalscalarfunction.First,let’stakealookathowtoreturnasetofintegers.

ReturningasetofintegersWewillrevisitourFibonaccinumbergeneratingfunction;however,thistimewewillnotjustreturnthenthnumber,butthewholesequenceofnumbersuptothenthnumber,asshownhere:

CREATEORREPLACEFUNCTIONfibonacci_seq(numinteger)

RETURNSSETOFintegerAS$$

DECLARE

aint:=0;

bint:=1;

BEGIN

IF(num<=0)

THENRETURN;

ENDIF;

RETURNNEXTa;

LOOP

EXITWHENnum<=1;

RETURNNEXTb;

num=num-1;

SELECTb,a+bINTOa,b;

ENDLOOP;

END;

$$LANGUAGEplpgsql;

Thefirstdifferencewesee,isthatinsteadofreturningasingleintegervalue,thisfunctionisdefinedtoreturnaSETOFinteger.

Then,ifyouexaminethecodecarefully,youwillseethattherearetwodifferenttypesofRETURNstatements.ThefirstistheordinaryRETURNfunctioninthefollowingcodesnippet:

IF(num<=0)

THENRETURN;

Inthiscase,theIFfunctionisusedtoterminatethefibonacci_seqfunctionearly,ifthelengthofthedesiredsequenceofFibonaccinumbersiszeroorless.

ThesecondkindofRETURNstatementisusedtoreturnvaluesandcontinueexecution:

RETURNNEXTa;

RETURNNEXTappendsrowstotheresultsetofthefunction,andtheexecutioncontinuesuntilanormalRETURNstatementisencounteredoruntilthecontrolreachestheendofthefunction.YoumayhavenoticedthatthereareafewotherthingswediddifferentlyinthisFibonacciexample,thanwedidearlier.First,wedeclaredandinitializedthevariablesaandbinsidetheDECLAREsection,insteadoffirstdeclaringandtheninitializingthem.Wealsousedtheargumentasadowncounterinsteadofusingaseparatevariabletocountfromzeroandthencomparingitwiththeargument.

Let’stestourfunctionnow.Inthenextsection,wewilldiscusshowtousethesetreturningfunctionsinmoredetail:

postgres=#SELECTfibonacci_seq(5);

fibonacci_seq

---------------

0

1

1

2

3

(5rows)

Bothofthesetechniquessaveafewlinesofcodeandcanmakethecodemorereadable,dependingonyourpreferences.However,thelongerversionsmightbeeasiertofollowandunderstand,sowedon’tparticularlyendorseeitherway.

UsingasetreturningfunctionAsetreturningfunction(alsoknownasatablefunction)canbeusedinmostplaceswhereatable,view,orsubquerycanbeused.Theyareapowerfulandflexiblewaytoreturndata.

YoucancallthefunctionintheSELECTclause,asyoudowithascalarfunction:

postgres=#SELECTfibonacci_seq(3);

fibonacci_seq

---------------

0

1

1

(3rows)

YoucanalsocallthefunctionaspartoftheFROMclause:

postgres=#SELECT*FROMfibonacci_seq(3);

fibonacci_seq

---------------

0

1

1

(3rows)

YoucanevencallthefunctionintheWHEREclause:

postgres=#SELECT*FROMfibonacci_seq(3)WHERE1=ANY(SELECT

fibonacci_seq(3));

fibonacci_seq

---------------

0

1

1

(3rows)

Youcanlimittheresultset,justasinthecaseofqueryingatable:

postgres=#SELECT*FROMfibonacci_seq(10)asfibWHEREfib>3;

fibonacci_seq

---------------

5

8

13

21

34

(5rows)

Usingdatabase-sidefunctionsforallthedataaccessisagreatwaytosecureyourapplication;italsohelpswithperformanceandallowseasymaintenance.Tablefunctionsallowyoutousefunctionsinallcaseswhereyouwouldhavebeenforcedtousemorecomplexqueriesfromtheclientifonlyscalarfunctionswereavailable.

Returningrowsfromafunctionwilloftenbehelpfultoreturnbacktotheclientmore

informationthanjustasetofintegers.Youmayneedallthecolumnsfromanexistingtable,andthesimplestwaytodeclareareturntypeforafunctionistojustusethetableaspartofthereturndefinition,asshownhere:

CREATEORREPLACEFUNCTIONinstalled_languages()

RETURNSSETOFpg_languageAS$$

BEGIN

RETURNQUERYSELECT*FROMpg_language;

END;

$$LANGUAGEplpgsql;

NoticethatyoustillneedtheSETOFpart,butinsteadofdefiningitasasetofintegers,weusepg_language,whichisatable.

YoucanuseTYPE,definedusingtheCREATETYPEcommandorevenVIEW:

hannu=#SELECT*FROMinstalled_languages();

-[RECORD1]-+----------

lanname|internal

lanowner|10

lanispl|f

lanpltrusted|f

lanplcallfoid|0

laninline|0

lanvalidator|2246

lanacl|

-[RECORD2]-+----------

lanname|c

lanowner|10

lanispl|f

lanpltrusted|f

lanplcallfoid|0

laninline|0

lanvalidator|2247

lanacl|

-[RECORD3]-+----------

lanname|sql

lanowner|10

lanispl|f

lanpltrusted|t

lanplcallfoid|0

laninline|0

lanvalidator|2248

lanacl|

-[RECORD4]-+----------

lanname|plpgsql

lanowner|10

lanispl|t

lanpltrusted|t

lanplcallfoid|12596

laninline|12597

lanvalidator|12598

lanacl|

-[RECORD5]-+----------

lanname|plpythonu

lanowner|10

lanispl|t

lanpltrusted|f

lanplcallfoid|17563

laninline|17564

lanvalidator|17565

lanacl|

FunctionsbasedonviewsCreatingafunctionbasedonaviewdefinitionisaverypowerfulandflexiblewayofprovidinginformationtousers.Asanexampleofthis,IwilltellyouastoryofhowIstartedasimpleutilityviewtoanswerthequestion,“Whatqueriesarerunningnow,andwhichquerieshavebeenrunningforthelongesttime?”Thisevolvedintoafunctionbasedonthisview,plusafewmoreviewsbasedonthefunction.

ThewaytogetallthedatatoanswerthisquestioninPostgreSQLisbyusingthefollowingquery.Pleasenotethattheoutputisusinganexpandedmodeofpsql.Youcanturnitonusingthe\xmeta-command:

hannu=#SELECT*FROMpg_stat_activityWHEREstate='active';

-[RECORD1]----+--------------------------------

datid|17557

datname|hannu

pid|8933

usesysid|10

usename|postgres

application_name|psql

client_addr|

client_hostname|

client_port|-1

backend_start|2013-03-1913:47:45.920902-04

xact_start|2013-03-1914:05:47.91225-04

query_start|2013-03-1914:05:47.91225-04

state_change|2013-03-1914:05:47.912253-04

waiting|f

state|active

query|select*frompg_stat_activity|wherestate='active';

Theusualprocessistouseavariantofthefollowingquery,whichisalreadywrappedintoaviewhere:

CREATEVIEWrunning_queriesAS

SELECT

(CURRENT_TIMESTAMP-query_start)asruntime,

pid,

usename,

waiting,

query

FROMpg_stat_activity

WHEREstate='active'

ORDERBY1DESC

LIMIT10;

Soon,youwillnoticethatputtingthisqueryintoaviewisnotenough.Sometimes,youmaywanttovarythenumberoflowestqueries,whilesometimesyoumaynotwanttohavethefullquerytext,butjustthebeginning,andsoon.

Ifyouwanttovarysomeparameters,thelogicalthingtodoistouseafunctioninsteadofaview,asotherwiseyouwillneedtocreatedifferentviewsofeachnewrequirement:

CREATEORREPLACEFUNCTIONrunning_queries(rowsint,qlenint)

RETURNSSETOFrunning_queriesAS

$$

BEGIN

RETURNQUERYSELECT

runtime,

pid,

usename,

waiting,

substring(query,1,qlen)asquery

FROMrunning_queries

ORDERBY1DESC

LIMITrows;

END;

$$LANGUAGEplpgsql;

Asasecurityprecaution,thedefaultbehaviorofthepg_stat_activityviewisthatonlysuperuserscanseewhatotherusersarerunning.Sometimes,itmaybenecessarytoallowthenon-superuserstoatleastseethetypeofquery(SELECT,INSERT,DELETE,orUPDATE)thatotherusersarerunning,buthidetheexactcontents.Todothis,youhavetomaketwochangestothepreviousfunction.

First,replacetherowtogetcurrent_querywiththefollowingcodesnippet:

(CASEWHEN(usename=session_user)

OR(SELECTusesuper

FROMpg_user

WHEREusename=session_user)

THEN

substring(query,1,qlen)

ELSE

substring(ltrim(query),1,6)||'***'

END)asquery

Thiscodesnippetcheckseachrowtoseewhethertheuserrunningthefunctionhaspermissiontoseethefullquery.Iftheuserisasuperuser,thenheshehaspermissiontoseethefullquery.Iftheuserisaregularuser,he/shewillonlyseethefullqueryforhis/herqueries.Allotherrowswillonlyshowthefirstsixcharactersfollowedby***tomarkitasashortenedquerystring.

Theotherkeypointtoallowingordinaryuserstorunthefunction,istograntthemtheappropriaterightstodoso.Whenafunctioniscreated,thedefaultbehavioristorunwiththeSECURITYINVOKERrights,whichmeansthatthefunctionwillbecalledwiththerightsoftheuserwhocalledit.Toeasilygrantthecorrectrightstocallthefunction,thefunctionneedstobecreatedwiththeSECURITYDEFINERattribute.Thiscausesthefunctiontoexecutewiththeprivilegesoftheuserwhocreatedthefunction;therefore,creatingthefunctionasasuperuserwillallowittoexecuteasasuperuser,regardlessoftherightsoftheuserwhocalledit.This,however,shouldbedonewithcautionbecauseasuperusercanbedangerous,andifyouendupexecutingastringthatispassedin,youwillhavealltheissuesofSQLinjectionattacks.

Now,youhaveafunction,whichyoucanusetogetthestartofthefivelongest-running

queriesusingthefollowingquery:

SELECT*FROMrunning_queries(5,25);

Inordertogetacompletequery,youcanusethefollowing:


Youmaywanttodefineafewconvenienceviewsforthevariantsyouusemost,asfollows:

CREATEORREPLACEVIEWrunning_queries_tinyAS


CREATEVIEWrunning_queries_fullAS


Youcanevenredefinetheoriginalviewtousethefirstversionofthefunction:

CREATEORREPLACEVIEWrunning_queriesAS


Thisisusuallynotrecommended,butitdemonstratesthefollowingthreeimportantthings:

ViewsandfunctionscanhaveexactlythesamenameYoucangetacircularreferencebybasingafunctiononaviewandthenbasingaviewonthatfunctionIfyougetacircularreferenceinthisway,youcan’teasilychangeeitherdefinition

Toresolvethis,simplyavoidcircularreferences.

Evenwithoutcircularreferences,thereisstilladependencyontheviewcalledfromthefunction.If,forinstance,youneedtoaddacolumntoshowtheapplicationnametotherunning_queriesview,thefunctionneedstochangeaswell:

CREATEORREPLACEVIEWrunning_queriesAS

SELECT

CURRENT_TIMESTAMP-query_startasruntime,

pid,

usename,

waiting,

query,

application_nameasappname

FROMpg_stat_activity

ORDERBY1DESC

LIMIT10;

Theviewdefinitioncanbechangedwithoutanerror,butthenexttimeyoutrytoruntherunning_queries(int,int)function,youwillgetanerror:

hannu=#SELECT*FROMrunning_queries(5,25);

ERROR:structureofquerydoesnotmatchfunctionresulttype

DETAIL:Numberofreturnedcolumns(5)doesnotmatchexpectedcolumn

count(6).

CONTEXT:PL/pgSQLfunction"running_queries"line3atRETURNQUERY

Tofixthis,youneedtoaddanadditionalcolumntothefunction.Thisisoneofthe

dangersofreusingtypesinthisway:youmightendupbreakingfunctionsunintentionally.PostgreSQLwon’ttellyou,whenyouchangeatypeinthisway,whetheranyfunctionswereusingthistypeandwillonlyfailwhenyoutrytorunthequery:

CREATEORREPLACEFUNCTIONrunning_queries(rowsint,qlenint)

RETURNSSETOFrunning_queriesAS

$$

BEGIN

RETURNQUERYSELECT

runtime,

pid,

usename,

waiting,

(CASEWHEN(usename=session_user)

OR(selectusesuper

frompg_user

whereusename=session_user)

THEN

substring(query,1,qlen)

ELSE

substring(ltrim(query),1,6)||'***'

END)asquery,

appname

FROMrunning_queries

ORDERBY1DESC

LIMITrows;

END;

$$

LANGUAGEplpgsql

SECURITYDEFINER;

OUTparametersandrecordsUsingapre-existingtype,table,orviewforcompoundreturntypesisasimplemechanismforreturningmorecomplexstructures.However,thereisoftenaneedtodefinethereturntypeofthefunctionwiththefunctionitselfandnotdependonotherobjects.Thisisespeciallytruewhenmanagingchangestoarunningapplication;so,overaperiodoftime,twobetterwaystohandlethishavebeenaddedtoPostgreSQL.

OUTparametersUntilnow,allthefunctionswecreatedusedparametersthataredefinedasINparameters.TheINparametersaremeanttojustpassinformationintothefunctionthatcanbeused,butnotreturned.ParameterscanalsobedefinedasOUTorINOUTparametersifyouwantthefunctiontoreturnsomeinformationaswell:

CREATEORREPLACEFUNCTIONpositives(

INOUTaint,

INOUTbint,

INOUTcint)

AS$$

BEGIN

IFa<0THENa=null;ENDIF;

IFb<0THENb=null;ENDIF;

IFc<0THENc=null;ENDIF;

END;

$$LANGUAGEplpgsql;

Whenwerunthepreviousfunction,itonlyreturnsasinglerowofdatawiththreecolumns,asshownhere:

hannu=#SELECT*FROMpositives(-1,1,2);

-[RECORD1]

a|

b|1

c|2

ReturningrecordsIfmultiplerowsofdataneedtobereturned,asimilarfunctionthatreturnsasetisachievedbyaddingRETURNSSETOFRECORD.ThistechniquecanonlybeusedwithfunctionsusingtheINOUTorOUTarguments:

CREATEFUNCTIONpermutations(INOUTaint,

INOUTbint,

INOUTcint)

RETURNSSETOFRECORD

AS$$

BEGIN

RETURNNEXT;

SELECTb,cINTOc,b;RETURNNEXT;

SELECTa,bINTOb,a;RETURNNEXT;


SELECTa,bINTOb,a;RETURNNEXT;


END;

$$LANGUAGEplpgsql;

Runningthepermutationsfunctionreturnssixrows,asexpected:

hannu=#SELECT*FROMpermutations(1,2,3);

-[RECORD1]

a|1

b|2

c|3

-[RECORD2]

a|1

b|3

c|2

-[RECORD3]

a|3

b|1

c|2

-[RECORD4]

a|3

b|2

c|1

-[RECORD5]

a|2

b|3

c|1

-[RECORD6]

a|2

b|1

c|3

Thisworkswell,butitseemsabitverboseforwhatisaprettysimpleoperation.ThisisbecausewecannotdirectlycallRETURNNEXTa,b,c,butweneedtofirstassignvaluestovariablesdeclaredbytheINOUTincantations.Wealsowanttoavoidtheevenclumsiersyntaxoftmp=a;a=b;b=tmp;.

DuetodesigndecisionsinthePL/pgSQLlanguage,thereiscurrentlynogoodwayto

constructthereturnstructureatruntime,thatis,noRETURNa,b,c.

However,let’strytodoitanywayandseewhathappens.

UsingRETURNSTABLEYoumaythinkthatiftherearenovisibleOUTparametersinafunctiondeclaredasRETURNSTABLE(...),thefollowingcodemightwork:

CREATEFUNCTIONpermutations2(aint,bint,cint)RETURNSTABLE(aint,b

int,cint)

AS$$

BEGIN

RETURNNEXTa,b,c;

END;

$$LANGUAGEplpgsql;

However,whenwetrytodoitthisway,wegetanerror:

ERROR:parametername"a"usedmorethanonce

CONTEXT:compilationofPL/pgSQLfunction"permutations2"nearline1

ThiserrorhintsthatthefieldsinthereturntabledefinitionarealsoactuallyjustOUTparametersandthewholeRETURNSTABLEsyntaxisjustanotherwaytospellCREATEFUNCTIONf(OUT…,OUT…)RETURNSRECORD….

Thiscanbeverifiedfurtherbychangingtheinputparameters,sothatthedefinitioncanbefedintoPostgreSQL:

CREATEFUNCTIONpermutations2(iaint,ibint,icint)

RETURNSTABLE(aint,bint,cint)

AS$$

BEGIN

RETURNNEXTa,b,c;

END;

$$LANGUAGEplpgsql;

Whenwetrytocreatethefunction,wegetthefollowingoutput:

ERROR:RETURNNEXTcannothaveaparameterinfunctionwithOUTparameters

LINE5:RETURNNEXTa,b,c;

^

Soyes,thefieldsofthetableintheRETURNSdefinitionareactuallyjustOUTparameters.

ReturningwithnopredefinedstructureSometimes,youreallyneedtowriteafunctionwherethereturnstructureisunknown.OnegoodthingaboutPostgreSQLfunctiondeclarations,isthatyoucanusethereturntypeRECORD,whichcanbeleftundefineduntilthefunctioniscalled:

CREATEORREPLACEFUNCTIONrun_a_query(queryTEXT)

RETURNSSETOFRECORD

AS$$

DECLARE

retvalRECORD;

BEGIN

FORretvalINEXECUTEqueryLOOP

RETURNNEXTretval;

ENDLOOP;

END;

$$LANGUAGEplpgsql;

Thisisafunctionthatletsauserexecuteaquery;itisquiteuselessassuch,butitcanbeusedasthebasisformoreusefulfunctionsthat,forexample,letusersrunqueriesonlyatacertaintime,orcanbeusedtoperformsomechecksonqueriesbeforerunningthem.

Simplyrunthefollowingquery:

SELECT*FROMrun_a_query('SELECTusename,usesysidFROMpg_user');

Youwillgetthefollowingerror:

ERROR:acolumndefinitionlistisrequiredforfunctionsreturning

"record"

LINE1:select*fromrun_a_query('selectusename,usesysidfrompg_…

Tousethiskindofafunction,youneedtotellPostgreSQLwhatthereturnvalueswillbe,byaddingacolumndefinitionlistatthetimeofcallingafunctioninthefollowingway:

SELECT*FROMrun_a_query('SELECTusename,usesysidFROMpg_user')AS

("user"text,uidint);

So,willthiswork?No,youwillgetthefollowingerror:

ERROR:wrongrecordtypesuppliedinRETURNNEXT

DETAIL:Returnedtypenamedoesnotmatchexpectedtypetextincolumn1.

CONTEXT:PL/pgSQLfunctionrun_a_query(text)line6atRETURNNEXT

Bychangingthingsslightly,wefinallyarriveatsomethingthatworks,asfollows:

hannu=#SELECT*FROMrun_a_query('SELECTusename::text,usesysid::intFROM

pg_user')AS("user"text,uidint);

-[RECORD1]--

user|postgres

uid|10

-[RECORD2]--

user|hannu

uid|17573

Whatdowelearnfromthis?PostgreSQLwillletyoureturnanarbitraryrecordfroma

function,butitisveryparticularinhowitdoesthis.Whenyoucallthefunction,youwillneedtobeverydeliberateaboutthings,especiallydatatypes.PostgreSQLwillusedefaultcaststoconvertdatatodifferentdatatypesifithasenoughinformation.However,inafunctionsuchasthis,muchofthatinformationisnotknown.

ReturningSETOFANYThereisanotherwaytodefinefunctionsthatcanoperateon,andreturn,incompletetypedefinitions:usingtheANY*pseudotypes.

Let’sdefineafunction,whichturnsasimpleone-dimensionalPostgreSQLarrayofanytypeintoasetofrowswithoneelementofthesametype:

CREATEORREPLACEFUNCTIONarray_to_rows(array_inANYARRAY)

RETURNSTABLE(row_outANYELEMENT)

AS$$

BEGIN

FORiIN1..array_upper(array_in,1)LOOP

row_out=array_in[i];

RETURNNEXT;

ENDLOOP;

END;

$$LANGUAGEplpgsql;

Thisworkswellonanarrayofintegers:

hannu=#SELECTarray_to_rows('{1,2,3}'::int[]);

-[RECORD1]-+--

array_to_rows|1

-[RECORD2]-+--

array_to_rows|2

-[RECORD3]-+--

array_to_rows|3

Italsoworkswellonanarrayofdates,asshownhere:

hannu=#SELECTarray_to_rows('{"1970-1-1","2012-12-12"}'::date[]);

-[RECORD1]-+-----------

array_to_rows|1970-01-01

-[RECORD2]-+-----------

array_to_rows|2012-12-12

Thefunctionevenworksonarraysofrecordsfromuser-definedtables:

hannu=#CREATETABLEmydata(idserialprimarykey,datatext);

CREATETABLE

hannu=#INSERTINTOmydataVALUES(1,'one'),(2,'two');

INSERT02

hannu=#SELECTarray_to_rows(array(SELECTmFROMmydatam));

-[RECORD1]-+--------

array_to_rows|(1,one)

-[RECORD2]-+--------

array_to_rows|(2,two)

hannu=#SELECT*FROMarray_to_rows(array(SELECTmFROMmydatam));

-[RECORD1]

id|1

data|one

-[RECORD2]

id|2

data|two

ThelasttwoSELECTstatementsreturnaone-columntableofthetypemydataandatwo-columntableofthesametypeexpandedintoitscomponentcolumns.Thissinglefunctionisflexibleenoughtohandleseveraldifferenttypesofdatawithoutanychanges.

NoteThereisamorepotentversionofarray_to_rowsbuiltintoPostgreSQLcalledunnest().Thebuilt-infunctionisfasterthanoursamplefunctionandcanalsodealwitharrayswithmorethanonedimension:

hannu=#SELECTunnest('{{1,2,3},{4,5,6}}'::int[]);

-[RECORD1]

unnest|1

-[RECORD2]

unnest|2

-[RECORD3]

unnest|3

-[RECORD4]

unnest|4

-[RECORD5]

unnest|5

-[RECORD6]

unnest|6

PostgreSQLhasaweirdarraytype,whichcanholdthearraysofanynumberofdimensions.Itisevenweirderthanthis,asthearrayslicesinanydimensioncanalsostartwithanypositiveindex(andtheyare,bydefault,1-based).Forexample,anarraywithindicesrangingfrom-2to2isproducedbythefollowingincantation:

hannu=#SELECT'[-2:2]={1,2,3,4,5}'::int[];

-[RECORD1]------------

int4|[-2:2]={1,2,3,4,5}

Tocheckwhetherthisreallyisthecase,usethefollowingcodesnippet:

hannu=#SELECTarray_dims('[-2:2]={1,2,3,4,5}'::int[]);

-[RECORD1]------

array_dims|[-2:2]

Thethirdelementofthearrayis3,andthisisthemiddleelement.

VariadicargumentlistsPostgreSQLallowsyoutowriteafunctionwithavariablenumberofarguments.ThisisaccomplishedusingVARIADIC.Let’stakealookatapracticalexample.Supposeyouwanttolimittheresultsofaqueryinyourfunction,basedontheVARIADIClistofarguments;here’sonewaytodothis:

CREATEORREPLACEFUNCTIONget_nspc_tbls(VARIADICarrname[])

RETURNSTABLE(table_namename,idoid,nspnamename)

AS$$

BEGIN

RETURNQUERYSELECTc.relname,c.oid,n.nspnamefrompg_classc,

pg_namespacenwherec.relnamespace=n.oidandn.nspname=any(arr);

END;

$$LANGUAGEplpgsql;

Theprecedingfunctionletsyoulistthetables,whichareinspecifiednamespaces.Thevariablelistallowsyoutoprovideoneormorenamespacenames.Thistrickcanbequitehandyinvarioussituations.Noticetheuseoftheanyfunction;youcan’tsubstituteitwithanINclause,asthiswilltrytocompareanametypewithname[]:

postgres=#SELECT*FROMget_nspc_tbls('public','pg_temp');

-[RECORD1]------------------------

table_name|a

id|16434

nspname|public

-[RECORD2]------------------------

table_name|parameter

id|24682

nspname|public

-[RECORD3]------------------------

table_name|application_settings_old

id|24690

nspname|public

-[RECORD4]------------------------

table_name|foo

id|16455

nspname|pg_temp

AsummaryoftheRETURNSETOFvariantsYoulearnedthatyoucanreturntable-likedatasetsfromafunctionusingoneofthefollowing:

RETURNS RECORDstructure INSIDEfunction

SETOF<type> Thisisobtainedfromthetypedefinition

DECLARErowvariableoftheROWorRECORDtype

ASSIGNtorowvariable

RETURNNEXTvar;

SETOF

<table/view>Thisisthesameasthetableorviewstructure

SETOFRECORD DynamicusingAS(nametype,…)atcallsite

SETOFRECORD ThisusestheOUTandINOUTfunctionarguments.AssignedtotheOUTvariables RETURNNEXT;

TABLE(...)

Thisisdeclaredinline,inparentheses,aftertheTABLEkeywordandisconvertedtotheOUTvariableforuseinfunctions.ItisassignedtotheOUTvariablesfromtheTABLE(...)partofthedeclaration.

RETURNNEXT;

ReturningcursorsAnothermethodthatcanbeusedtogettabulardataoutofafunction,isusingCURSOR.

CURSOR,oraportal,asitissometimesreferredtoinPostgreSQLdocumentation,isaninternalstructurethatcontainsapreparedqueryplan,readytoreturnrowsfromthequery.Sometimes,thecursorneedstoretrieveallthedataforthequeryatonce,butformanyqueriesitdoeslazyfetching.Forexample,queriesthatneedtoscanallofthedatainatable,suchasSELECT*FROMxtable,onlyreadtheamountofdatathatisneededforeachFETCHfromthecursor.

InplainSQL,CURSORisdefinedasfollows:

DECLAREmycursorCURSORFOR<query>;

Later,therowsarefetchedusingthefollowingstatement:

FETCHNEXTFROMmycursor;

Whileyoucanuseacursortohandlethedatafromasetreturningfunctiontheusualway,bysimplydeclaringthecursorasDECLAREmycursorCURSORFORSELECT*FROMmysetfunc();,manytimesitismorebeneficialtohavethefunctionitselfjustreturnacursor.

Youwillwanttodothisifyouneeddifferentcursorsbasedonargumentvalues,orifyouneedtoreturndynamicallystructureddataoutofafunction,withoutdefiningthestructurewhencallingthefunction.

ThecursorinPL/pgSQLisrepresentedbyavariableofthetyperefcursorandmustbedeclaredinoneofthefollowingthreeways:

DECLARE

curs1refcursor;

curs2CURSORFORSELECT*FROMtenk1;

curs3CURSOR(keyinteger)ISSELECT*FROMtenk1WHEREunique1=key;

ThefirstvariantdeclaresanunboundcursorthatneedstobeboundtoaqueryatOPENtime.Thetworemainingvariantsdeclareacursorboundtoaquery.

NoteYoucanreadagoodtechnicaloverviewonhowtousecursorsinPL/pgSQLfunctionsfromtheofficialPostgreSQLdocumentationathttp://www.postgresql.org/docs/current/static/plpgsql-cursors.html.

Onethingtonoteaboutthedocumentationisthatyoudon’treallyneedto“return”thecursor,atleastnotnowbecausecursorscanalsobepassedbacktothecallerinOUTparameters.

ThePostgreSQLdocumentationstates:

“Thefollowingexampleshowsonewaytoreturnmultiplecursorsfromasingle

http://www.postgresql.org/docs/current/static/plpgsql-cursors.html

function:

CREATEFUNCTIONmyfunc(refcursor,refcursor)RETURNSSETOFrefcursorAS$$

BEGIN

OPEN$1FORSELECT*FROMtable_1;

RETURNNEXT$1;

OPEN$2FORSELECT*FROMtable_2;

RETURNNEXT$2;

END;

$$LANGUAGEplpgsql;

—needtobeinatransactiontousecursors.

BEGIN;

SELECT*FROMmyfunc('a','b');

FETCHALLFROMa;

FETCHALLFROMb;

COMMIT;"

YoucanalsowritethemyfuncfunctionusingtheOUTparameters:

CREATEFUNCTIONmyfunc2(cur1refcursor,cur2refcursor)

RETURNSVOIDAS$$

BEGIN

OPENcur1FORSELECT*FROMtable_1;

OPENcur2FORSELECT*FROMtable_2;

END;

$$LANGUAGEplpgsql;

Youwillstillrunthefunctioninexactlythesamewayasthefunctionreturningthecursorvariable.

IteratingovercursorsreturnedfromanotherfunctionTowrapupourcursors’discussion,let’sgothroughanexampleofreturningacursorandtheniteratingoverthereturnedcursorinanotherPL/pgSQLfunction:

1. First,let’screateafive-rowtableandfillitwithdata:

CREATETABLEfiverows(idserialPRIMARYKEY,datatext);

INSERTINTOfiverows(data)VALUES('one'),('two'),

('three'),('four'),('five');

2. Next,let’sdefineourcursorreturningfunction.Thisfunctionwillopenacursorforaquery,basedonitsargumentandthenreturnthatcursor:

CREATEFUNCTIONcurtest1(currefcursor,tagtext)

RETURNSrefcursor

AS$$

BEGIN

OPENcurFORSELECTid,data||'+'||tagFROMfiverows;

RETURNcur;

END;

$$LANGUAGEplpgsql;

3. Next,wedefineafunction,whichusesthefunctionwejustcreatedtoopentwoadditionalcursors,andthenprocessthequeryresults.Toshowthatwearenotcheatingandthatthefunctionreallycreatesthecursors,weusethefunctiontwiceanditerateovertheresultsinparallel:

CREATEFUNCTIONcurtest2(tag1text,tag2text)

RETURNSSETOFfiverows

AS$$

DECLARE

cur1refcursor;

cur2refcursor;

rowrecord;

BEGIN

cur1=curtest1(NULL,tag1);

cur2=curtest1(NULL,tag2);

LOOP

FETCHcur1INTOrow;

EXITWHENNOTFOUND;

RETURNNEXTrow;

FETCHcur2INTOrow;

EXITWHENNOTFOUND;

RETURNNEXTrow;

ENDLOOP;

END;

$$LANGUAGEplpgsql;

Pleasenote,thatoncearecordvariableinsideaplpgsqlfunctionisdefinedbybeingused,itcan’tbechangedfromrecordtypeforthedurationofthatsessionbecause

PL/pgSQLstoresandreusestheplanbuiltwiththerecordtype.

BypassinginNULLtothefirstparametersofcurtest1,PostgreSQLautomaticallygeneratesthecursornamessothatmultipleinvocationsofthisfunctionwillnotgetnameconflictswithanyotherfunctions,whichalsocreatecursors.

WrappingupoffunctionsreturningcursorsTheprosofusingcursorsareasfollows:

Cursorsareausefultoolifyoudon’twanttoalwaysexecutethequeryandwaitforthefullresultsetbeforereturningfromafunctionCurrently,theyarealsotheonlywaytoreturnmultipleresultsetsoutofauser-definedfunction

Theconsofusingcursorsareasfollows:

Theymainlyworktopassdatabetweenfunctionsontheserver,andyouarestilllimitedtoonerecordsetpercallreturnedtothedatabaseclientTheyaresometimesconfusingtouse,andboundandunboundcursorsarenotalwaysinterchangeable

NoteYoucanreadmoreaboutcursorsathttp://www.postgresql.org/docs/current/static/plpgsql-cursors.html.

http://www.postgresql.org/docs/current/static/plpgsql-cursors.html

OtherwaystoworkwithstructureddataWenowhavecoveredthetraditionalwaysofreturningsetsofstructureddatafromfunctions.Wewillnowstartwiththemoreinterestingpart.Othermethodstopassaroundcomplexdatastructureshaveevolvedintheworld.

Complexdatatypesforthemodernworld–XMLandJSONIntherealworld,mostofthedataisnotinasingletableandthedatabaseisnotthemainthingthatmostprogrammersfocuson.Often,theydon’teventhinkaboutitatall,oratleasttheywouldrathernotthinkaboutit.

Ifyouareadatabasedeveloperwhoworksonthedatabasesideofthings,itisoftendesirabletotalktotheclients(beitweborapplicationdevelopersasyourclient,orprogramsasdatabaseclients)inthelanguagetheyspeak.Currently,thetwomostwidelyspokendatalanguagesbythewebapplicationsandtheirdevelopersareXMLandJSON.

BothXMLandJSONaretext-baseddataformats,andassuch,theycanbeeasilysavedintofieldsofthetypetext.PostgreSQL,beingaDBMS,builtforbeinguser-extendable,alsohasextensivesupportforboththeseformats.

XMLdatatypeandreturningdataasXMLfromfunctionsOneoftheextensionsaddedtoPostgreSQL,inordertosupportXMLdata,isanativeXMLdatatype.WhiletheXMLdatatypeislargelyjustatextfield,itdiffersfromtextinthefollowingways:

TheXMLstoredinanXMLfieldischeckedoninsertsandupdatestobewell-formedTherearesupportfunctionstoproduceandworkwithknownwell-formedXML

AnXMLvaluecanbeproducedinacoupleofways,includingthestandardSQLmethod:

XMLPARSE({DOCUMENT|CONTENT}value)

PostgreSQLalsohasaspecificsyntaxthatwillproduceanXMLvalue:

xml'<foo>bar</foo>'

'<foo>bar</foo>'::xml

AnXMLvaluecanbeeasilyconvertedtoatextrepresentationusingtheXMLSERIALIZEfunction,asfollows:

XMLSERIALIZE({DOCUMENT|CONTENT}valueAStype)

Additionally,PostgreSQLallowsyoutosimplycasttheXMLvalueastext.

NoteThefulldescriptionoftheXMLdatatypeanditsassociatedfunctionsisavailableathttp://www.postgresql.org/docs/current/static/datatype-xml.html.AseachversionofPostgreSQLhasimproved,thesupportforXMLhasalsoimproved.

Thereareseveral*_to_xmlfunctionsinPostgreSQL,whichtakeeitheraSQLquery,oratable,orviewasinputandreturnitscorrespondingXMLrepresentation.

Let’stakealookatthisusingthefiverowstablewedefinedpreviouslyintheReturningcursorssection.

First,let’sgetthetabledataasanXML:

hannu=#SELECTtable_to_xml('fiverows',true,false,'')ASs;

-[RECORD1]-------------------------------------------------------

s|<fiverowsxmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

|

|<row>

|<id>1</id>

|<data>one</data>

|</row>

|

|<row>

|<id>2</id>

|<data>two</data>

|</row>

http://www.postgresql.org/docs/current/static/datatype-xml.html

|

|<row>

|<id>3</id>

|<data>three</data>

|</row>

|

|<row>

|<id>4</id>

|<data>four</data>

|</row>

|

|<row>

|<id>5</id>

|<data>five</data>

|</row>

|

|</fiverows>

|

IfyouhaveaclientthatcanhandleXML,thenthe*_to_xmlfunctionscanbethebestwaytoreturncomplexdata.

Anothergoodthingaboutthe*_to_xmlfunctionsisthatyoucancreateafunctionwhichreturnsseveraldifferentXMLdocumentsinonegoand,thus,returnsdatarowswithdifferentstructures.Agoodexample,willbeapaymentorderandrows,wherethefirstrecordreturnedbythefunctionistheXMLfortheorderheaderfollowedbyoneormorerecordsofXMLfortheorderrows,allreturnedbythesamefunctioninonecall.

Thereare,atthetimeofwritingthisbook,fivevariantsofthe*_to_xmlfunctions:

cursor_to_xml(cursorrefcursor,countinteger,nullsbool,tableforestbool,

targetnstext)

query_to_xml(querytext,nullsbool,tableforestbool,targetnstext)

table_to_xml(tblregclass,nullsboolean,tableforestboolean,targetns

text)

schema_to_xml(schemaname,nullsboolean,tableforestboolean,targetns

text)

database_to_xml(nullsboolean,tableforestbool,targetnstext)

Thecursor_to_xml(...)function,whichiteratesoveranopencursor,isrecommendedforlargedatasets,asitcanconvertthedataintochunksofrowswithoutfirstaccumulatingallthedatainmemory.

ThenextthreefunctionsreturnastringthatrepresentsaSQLquery,atablename,oraschemaname,andreturnallthedatafromthenamedobject.Thetable_to_xml()functionalsoworksonviews.Then,thereisthedatabase_to_xml(...)function,whichconvertsthecurrentdatabasetoanXMLdocument.However,acommonresultofrunningitonaproductiondatabaseisanout-of-memoryerroriftheoutputistoobig:

hannu=#SELECTdatabase_to_xml(true,true,'n');

ERROR:outofmemory

DETAIL:Failedonrequestofsize1024.

ReturningdataintheJSONformatInPostgreSQL9.2,thereisanewnativedatatypeforJSONvalues.ThisnewsupportforJSONfollowedthesamegrowthpatternasXML.ItinitiallystartedwithtwofunctionstoconvertarraysandrecordstotheJSONformat,butinPostgreSQL9.3,therearemanynewfunctionsintroduced.ThefollowingisasummaryofimprovementsinthePostgreSQL9.3JSONsupport:

TherearededicatedJSONoperatorssuchas->,->>,and#>TherearetennewJSONfunctions

TheJSONparserisexposedasanAPI.TheAPIprovideshooksforthesignificantparserevent,suchasthebeginningandendofobjectsandarrays,andprovidingfunctionstohandlethesehooksallowsawidevarietyofJSONprocessingfunctionstobecreated.

ThehstoreextensionhastwonewJSON-relatedfunctions,hstore_to_json(hstore)andhstore_to_json_loose(hstore).

NoteYoucanreadaboutthenewJSONoperatorsandthefunctionsintheofficialdocumentationathttp://www.postgresql.org/docs/current/static/functions-json.html.

Therow_to_json(record,bool)functionisusedtoconvertanyrecordtoJSON,andarray_to_json(anyarray,bool)isusedtoconvertanyarraytoJSON.

Thefollowingaresomesimpleexamplesoftheusageofthesefunctions:

hannu=#SELECTarray_to_json(array[1,2,3]);

-[RECORD1]-+--------

array_to_json|[1,2,3]

hannu=#SELECT*FROMtest;

-[RECORD1]--------------------

id|1

data|0.26281

tstampt|2012-04-0513:21:03.235

-[RECORD2]--------------------

id|2

data|0.1574

tstampt|2012-04-0513:21:05.201

hannu=#SELECTrow_to_json(t)FROMtestt;

-[RECORD1]-----------------------------------------------------

row_to_json|{"id":1,"data":0.26281,"tstampt":"2012-04-0513:21:03.235"}

-[RECORD2]-----------------------------------------------------

row_to_json|{"id":2,"data":0.1574,"tstampt":"2012-04-0513:21:05.201"}

ThesefunctionsareveryusefulastheyenableustowritefunctionsreturningmuchmorecomplexdatathanwaspossibleusingthestandardRETURNSTABLEsyntax,buttherealpowerofthesefunctionscomesfrombeingabletoconvertarbitrarilycomplexrows.

Let’sfirstcreateasimpletablewithsomedata:

http://www.postgresql.org/docs/current/static/functions-json.html

CREATETABLEtest(

idserialPRIMARYKEY,

datatext,

tstamptimestampDEFAULTcurrent_timestamp

);

INSERTINTOtest(data)VALUES(random()),(random());

Now,let’screateanothertable,whichhasonecolumnwiththedatatypeoftheprevioustable,andinsertrowsfromthistableinthenewtable:

hannu=#CREATETABLEtest2(

hannu(#idserialPRIMARYKEY,

hannu(#data2test,

hannu(#tstamptimestampDEFAULTcurrent_timestamp

hannu(#);

hannu=#INSERTINTOtest2(data2)SELECTtestFROMtest;

INSERT02

hannu=#SELECT*FROMtest2;

-[RECORD1]------------------------------------

id|5

data2|(1,0.26281,"2012-04-0513:21:03.235204")

tstamp|2012-04-3015:42:11.757535

-[RECORD2]------------------------------------

id|6

data2|(2,0.15740,"2012-04-0513:21:05.2033")

tstamp|2012-04-3015:42:11.757535

Now,let’sseehowrow_to_json()handlesthistable:

hannu=#SELECTrow_to_json(t2,true)FROMtest2t2;

row_to_json

-----------------------------------------------------------

{"id":5,

"data2":{"id":1,"data":"0.26281",

"tstamp":"2012-04-0513:21:03.235204"},

"tstamp":"2012-04-3015:42:11.757535"}

{"id":6,

"data2":{"id":2,"data":"0.15740",

"tstamp":"2012-04-0513:21:05.2033"},

"tstamp":"2012-04-3015:42:11.757535"}

(2rows)

TheresultwasconvertedtoJSONwithoutanyproblems.

Justtobesure,let’spushthecomplexityupabitmoreandcreateatest3table,whichhasanarrayofthetable2rowsasitsdatavalue:

CREATETABLEtest3(

idserialPRIMARYKEY,

data3test2[],

tstamptimestampDEFAULTcurrent_timestamp

);

INSERTINTOtest3(data3)

SELECTarray(SELECTtest2FROMtest2);

Let’sseewhetherrow_to_jsonstillworks,asshownhere:

SELECTrow_to_json(t3,true)FROMtest3t3;

------------------------------------------

{"id":1,

"data3":[{"id":1,

"data2":{"id":1,

"data":"0.262814193032682",

"tstamp":"2012-04-0513:21:03.235204"},

"tstamp":"2012-04-0513:25:03.644497"

},

{"id":2,

"data2":{"id":2,

"data":"0.157406373415142",

"tstamp":"2012-04-0513:21:05.2033"},

"tstamp":"2012-04-0513:25:03.644497"

}

],

"tstamp":"2012-04-1614:40:15.795947"}

(1row)

Yes,itdoes!

Well,actuallyIhadtomanuallyformatitalittle,astheprettyprintflagtorow_to_json()onlyforksforthetoplevel,andthesecondrowoftheresult(theonethatfollows"data3")wasallinoneline.However,JSONitselfwascompletelyfunctional!

SummaryThemainpointsyoulearnedinthischapterarethatyoucanreturnmultiplerowsandreturndatausingOUTparameters,aswellasreturnmultiplerowsofcomplexdata,similartoaSELECTquery.Youalsolearnedthatseveralsetsoftablescanbereturned,andyoucanpossiblyhavethemevaluatedinalazymannerusingrefcursors.Moreover,youcanreturndataascomplexasyouwantusingeitherXMLorJSON.

So,therereallyareveryfewreasonsfornotusingdatabasefunctionsasyourmaininteractionmechanismwiththedatabase.Inthenextchapter,wewilllearnhowtocallfunctionswhendifferenttypesofeventsoccurinthedatabase.

Chapter5.PL/pgSQLTriggerFunctionsWhileitisgenerallyagoodpracticetokeeprelatedcodetogetherandavoidhiddenactionsaspartofthemainapplicationcodeworkflows,therearevalidcaseswhereitisagoodpracticetoaddsomekindofgeneralorcross-applicationfunctionalitytothedatabaseusingautomatedactions,whichhappeneachandeverytimeatableismodified.Thatis,actionsarepartofyourdatamodelandnotyourapplicationcodeandyouwanttobesurethatitisnotpossibletoforgetorbypassthem,whichissimilartohowconstraintsmakeitimpossibletoinsertinvaliddata.

Themethodusedtoaddautomatedfunctioncallstoatablemodifyingeventiscalledatrigger.Triggersareespeciallyusefulforthecaseswheretherearemultipledifferentclientapplications—possiblyfromdifferentsourcesandusingdifferentprogrammingstyles—accessingthesamedatausingmultipledifferentfunctionsorsimpleSQL.

Fromastandard’sperspective,triggersarestandardizedinSQL3(SQL1999).Triggersaregenerallydefinedusinganevent-condition-action(ECA)model.Eventsoccurinthedatabase,whichtriggertheinvocationofacertainfunctioniftheconditionsaresatisfied.Alldataactionsperformedbythetriggerexecutewithinthesametransactioninwhichthetriggerisexecuting.TriggerscannotcontaintransactioncontrolstatementssuchasCOMMITandROLLBACK.Triggerscanbestatementlevelorrowlevel(moreonthislaterinthechapter).

InPostgreSQL,atriggerisdefinedintwosteps:

1. DefineatriggerfunctionusingCREATEFUNCTION.2. BindthistriggerfunctiontoatableusingCREATETRIGGER.

Inthischapter,wewillcoverthefollowingtopics:

CreatingatriggerandatriggerfunctionTakingalookatsomeoftheusecasesoftriggers,suchasanaudittrigger,anddisallowingcertainoperationsusingtriggersDiscussingconditionaltriggersandtriggersonspecificfieldchangesDescribingthelistofvariablespassedontoatriggerinbrief,whichcanbeusedinthetriggerfunctionforconditionallookups

CreatingthetriggerfunctionThetriggerfunction’sdefinitionlooksmostlylikeanordinaryfunctiondefinition,exceptthatithasareturnvaluetype,trigger,anditdoesnottakeanyarguments:

CREATEFUNCTIONmytriggerfunc()RETURNStriggerAS$$…

TriggerfunctionsarepassedinformationabouttheircallingenvironmentthroughaspecialTriggerDatastructure,whichinthecaseofPL/pgSQLisaccessiblethroughasetoflocalvariables.Thelocalvariables,OLDandNEW,representtherowinwhichthetriggerisinbeforeandaftertriggeringtheevent.Additionally,thereareseveralotherlocalvariablesthatstartwiththeprefixTG_,suchasTG_WHENorTG_TABLE_NAMEforgeneralcontext.Onceyourtriggerfunctionisdefined,youcanbindittoaspecificsetofactionsonatable.

CreatingthetriggerThesimplifiedsyntaxtocreateauser-definedTRIGGERstatementisgivenasfollows:

CREATETRIGGERname

{BEFORE|AFTER|INSTEADOF}{event[OR…]}

ONtable_name

[FOR[EACH]{ROW|STATEMENT}]

EXECUTEPROCEDUREfunction_name(arguments)

Intheprecedingcode,theeventiseitherINSERT,UPDATE,DELETE,orTRUNCATE.Thereareafewmoreoptionswhichwewillcomebacktoinalatersection.

Theargumentsvariableseeminglypassedtothetriggerfunctioninthetriggerdefinitionarenotusedasargumentswhencallingthetrigger.Instead,theyareavailabletothetriggerfunctionasatextarray(text[])intheTG_ARGVvariable(thelengthofthisarrayisinTG_NARGS).Let’sslowlystartinvestigatinghowtriggersandtriggerfunctionswork.

StartingfromPostgreSQL9.3,thereissupportforDDLtriggers.WewilllearnmoreaboutDDLtriggers,alsocalledeventtriggers,inthenextchapter.

First,wewilluseasimpletriggerexampleandmoveontomorecomplexexamplesstepbystep.

Workingonasimple“Hey,I’mcalled”triggerThefirsttriggerwewillworkonsimplysendsanoticebacktothedatabaseclienteachtimethetriggerisfiredandprovidessomefeedbackonitsfiringconditions:

CREATEORREPLACEFUNCTIONnotify_trigger()

RETURNSTRIGGERAS$$

BEGIN

RAISENOTICE'Hi,Igot%invokedFOR%%%on%',

TG_NAME,

TG_LEVEL,

TG_WHEN,

TG_OP,

TG_TABLE_NAME;

END;

$$LANGUAGEplpgsql;

Next,weneedatabletobindthisfunctiontothefollowinglineofcode:

CREATETABLEnotify_test(iint);

Nowwearereadytodefinethetrigger.Asthisisasimpleexample,wedefineatriggerwhichisinvokedonINSERTandcallsthefunctiononceoneachrow:

CREATETRIGGERnotify_insert_trigger

AFTERINSERTONnotify_test

FOREACHROW

EXECUTEPROCEDUREnotify_trigger();

Let’stestthisout:

postgres=#INSERTINTOnotify_testVALUES(1),(2);

NOTICE:Hi,Igotnotify_insert_triggerinvokedFORROWAFTERINSERTon

notify_test

ERROR:controlreachedendoftriggerprocedurewithoutRETURN

CONTEXT:PL/pgSQLfunctionnotify_trigger()

Hmm,itseemsweneedtoreturnsomethingfromthefunctioneventhoughitisnotneededforourpurposes.ThefunctiondefinitionsaysCREATEFUNCTION…RETURNSbutwedefinitelycannotreturnatriggerfromafunction.

Let’sgetbacktothedocumentation.OK,hereitis.ThetriggerneedstoreturnavalueoftheROWorRECORDtypeanditisignoredintheAFTERtriggers.

Fornow,let’sjustreturnNEWasthisistherighttypeandisalwayspresenteventhoughitwillbeNULLintheDELETEtrigger:

CREATEORREPLACEFUNCTIONnotify_trigger()

RETURNSTRIGGERAS$$

BEGIN

RAISENOTICE'Hi,Igot%invokedFOR%%%on%',

TG_NAME,

TG_LEVEL,TG_WHEN,TG_OP,TG_TABLE_NAME;

RETURNNEW;

END;

$$LANGUAGEplpgsql;

WecanuseRETURNNULLaswellhere,asthereturnvalueoftheAFTERtriggerisignoredanyway:

postgres=#INSERTINTOnotify_testVALUES(1),(2);


notify_test


notify_test

INSERT02

Asyousaw,thetriggerfunctionisindeedcalledonceforeachrowthatisinserted,solet’susethesamefunctiontoalsoreporttheUPDATEandDELETEfunctions:

CREATETRIGGERnotify_update_trigger

AFTERUPDATEONnotify_test

FOREACHROW


CREATETRIGGERnotify_delete_trigger

AFTERDELETEONnotify_test

FOREACHROW


Checkwhethertheprecedingcodeworks.

First,let’stesttheUPDATEtrigger:

postgres=#updatenotify_testseti=i*10;

NOTICE:Hi,Igotnotify_update_triggerinvokedFORROWAFTERUPDATEon

notify_test

NOTICE:Hi,Igotnotify_update_triggerinvokedFORROWAFTERUPDATEon

notify_test

UPDATE2

Thisworksfine—wegetanoticefortwoinvocationsofourtriggerfunction.

Now,let’stesttheDELETEtrigger:

postgres=#deletefromnotify_test;

NOTICE:Hi,Igotnotify_delete_triggerinvokedFORROWAFTERDELETEon

notify_test

NOTICE:Hi,Igotnotify_delete_triggerinvokedFORROWAFTERDELETEon

notify_test

DELETE2

Ifyouonlywanttobenotifiedeachtimeanoperationisperformedonthetable,theprecedingcodeisenough.Onesmallimprovementcanbemadeinhowwedefinetriggers.InsteadofcreatingonetriggerforeachoftheINSERT,UPDATE,orDELETEfunctions,wecancreateasingletriggertobecalledforanyofthem.So,let’sreplacethepreviousthreetriggerswithjustthefollowing:

CREATETRIGGERnotify_trigger

AFTERINSERTORUPDATEORDELETE

ONnotify_test

FOREACHROW


TheabilitytoputmorethanoneoftheINSERT,UPDATE,orDELETEfunctionsinthesametriggerdefinitionisaPostgreSQLextensiontotheSQLstandard.Sincetheactionpartofthetriggerdefinitionisnonstandardanyway,especiallywhenusingaPL/pgSQLtriggerfunction,thisshouldnotbeaproblem.

Let’snowdroptheindividualtriggers,truncatethetable,andtestagain:

postgres=#DROPTRIGGERnotify_insert_triggerONnotify_test;

DROPTRIGGER

postgres=#DROPTRIGGERnotify_update_triggerONnotify_test;

DROPTRIGGER

postgres=#DROPTRIGGERnotify_delete_triggerONnotify_test;

DROPTRIGGER

postgres=#TRUNCATEnotify_test;

TRUNCATETABLE

postgres=#INSERTINTOnotify_testVALUES(1);

NOTICE:Hi,Igotnotify_triggerinvokedFORROWAFTERINSERTon

notify_test

INSERT01

Thisworksfinebutitrevealsoneweakness:wedidnotgetanynotificationonTRUNCATE.

Unfortunately,wecannotsimplyaddORTRUNCATEintheprecedingtriggerdefinition.TheTRUNCATEcommanddoesnotactonsinglerows,sotheFOREACHROWtriggersmakenosensefortruncatingandarenotsupported.

YouneedtohaveaseparatetriggerdefinitionforTRUNCATE.Fortunately,wecanstillusethesamefunction,atleastforthissimple“Hey,I’mcalled”trigger:

CREATETRIGGERnotify_truncate_trigger

AFTERTRUNCATEONnotify_test

FOREACHSTATEMENT


NowwegetanotificationonTRUNCATEaswell,asshownhere:


NOTICE:Hi,Igotnotify_truncate_triggerinvokedFORSTATEMENTAFTER

TRUNCATEonnotify_test

TRUNCATETABLE

WhileitmayseemcooltogetthesemessagesineachDataManipulationLanguage(DML)operation,ithaslittleproductionvalue.

So,let’sdevelopthisabitfurtherandlogtheeventinanauditlogtableinsteadofsendingsomethingbacktotheuser.

TheaudittriggerOneofthemostcommonusesoftriggersistologdatachangestotablesinaconsistentandtransparentmanner.Whencreatinganaudittrigger,wefirstmustdecidewhatwewanttolog.

Alogicalsetofthingsthatcanbeloggedarewhochangedthedata,whenthedatawaschanged,andwhichoperationchangedthedata.Thisinformationcanbesavedinthefollowingtable:

CREATETABLEaudit_log(

usernametext,—whodidthechange

event_time_utctimestamp,—whentheeventwasrecorded

table_nametext,—containsschema-qualifiedtablename

operationtext,—INSERT,UPDATE,DELETEorTRUNCATE

before_valuejson,—theOLDtuplevalue

after_valuejson—theNEWtuplevalue

);

Here’ssomeadditionalinformationonwhatwewilllog:

TheusernamewillgettheSESSION_USERvariable,soweknowwhowasloggedinandnotwhichrolehehadpotentiallyassumedusingSETROLEevent_time_utcwillcontaintheeventtimeconvertedtoCoordinatedUniversalTime(UTC)sothatallcomplexdatearithmeticcalculationsarounddaylightsavingchangetimescanbeavoidedtable_namewillbeintheschema.tableformatTheoperationwillbedirectlyfromTG_OP,althoughitcouldbejustthefirstcharacter(I/U/D/T),withoutthelossofanyinformationFinally,thebeforeandafterimagesofrowsarestoredasrowsconvertedtojson,whichisavailableasitsowndatatypestartinginPostgreSQLVersion9.2foreasyhuman-readablerepresentationoftherowvalues

Next,thetriggerfunction:

CREATEORREPLACEFUNCTIONaudit_trigger()

RETURNStriggerAS$$

DECLARE

old_rowjson:=NULL;

new_rowjson:=NULL;

BEGIN

IFTG_OPIN('UPDATE','DELETE')THEN

old_row=row_to_json(OLD);

ENDIF;

IFTG_OPIN('INSERT','UPDATE')THEN

new_row=row_to_json(NEW);

ENDIF;

INSERTINTOaudit_log(

username,

event_time_utc,

table_name,

operation,

before_value,

after_value

)VALUES(

session_user,

current_timestampATTIMEZONE'UTC',

TG_TABLE_SCHEMA||'.'||TG_TABLE_NAME,

TG_OP,

old_row,

new_row

);

RETURNNEW;

END;

$$LANGUAGEplpgsql;

NoteTheconditionalexpressionsthatchecktheoperationsatthebeginningofthefunctionareneededtoovercomethefactthatNEWandOLDarenotNULLfortheDELETEandINSERTtriggerscorrespondingly.Rather,theyareunassigned.UsinganunassignedvariableinanyotherwayexceptassigningtoitinPL/pgSQLresultsinanerror.Anyerrorinthetriggerwillusuallyabortthetriggerandthecurrenttransaction.

Wearenowreadytodefineournewloggingtrigger,asshownhere:

CREATETRIGGERaudit_log

AFTERINSERTORUPDATEORDELETE

ONnotify_test

FOREACHROW

EXECUTEPROCEDUREaudit_trigger();

Let’srunasmalltest:weremoveouroriginalnotifytriggersfromthenotify_testtableandperformafewsimpleoperations:

postgres=#DROPTRIGGERnotify_triggerONnotify_test;

DROPTRIGGER

postgres=#DROPTRIGGERnotify_truncate_triggerONnotify_test;

DROPTRIGGER


TRUNCATETABLE

postgres=#INSERTINTOnotify_testVALUES(1);

INSERT01

postgres=#UPDATEnotify_testSETi=2;

UPDATE1

postgres=#DELETEFROMnotify_test;

DELETE1

postgres=#SELECT*FROMaudit_log;

-[RECORD1]--+---------------------------

username|postgres

event_time_utc|2013-04-1413:14:18.501529

table_name|public.notify_test

operation|INSERT

before_value|

after_value|{"i":1}

-[RECORD2]--+---------------------------

username|postgres

event_time_utc|2013-04-1413:14:18.51216


operation|UPDATE

before_value|{"i":1}

after_value|{"i":2}

-[RECORD3]--+---------------------------

username|postgres

event_time_utc|2013-04-1413:14:18.52331


operation|DELETE

before_value|{"i":2}

after_value|

Thisworkswell.Dependingonyourneeds,thisfunctionwilllikelyneedsometweaking.EnoughofjustwatchingandrecordingDML,itistimetostartinfluencingwhatgoesinthere.

NoteTriggersarecalledinalphabeticalorder,andthelattertriggerscanchangewhatgetsinserted,socautionneedstobeappliedwhennamingtriggers,particularlywhenauditing.

DisallowingDELETEWhatifourbusinessrequirementsaresuchthatthedatacanonlybeaddedandmodifiedinsometables,butnotdeleted?

OnewaytohandlethiswillbetojustrevoketheDELETErightsonthesetablesfromalltheusers(remembertoalsorevokeDELETEfromPUBLIC),butthiscanalsobeachievedusingtriggersbecauseofreasonssuchasauditingorreturningcustomexceptionmessages.

Agenericcanceltriggercanbewrittenasfollows:

CREATEORREPLACEFUNCTIONcancel_op()

RETURNSTRIGGERAS$$

BEGIN

IFTG_WHEN='AFTER'THEN

RAISEEXCEPTION'YOUARENOTALLOWEDTO%ROWSIN%.%',

TG_OP,TG_TABLE_SCHEMA,TG_TABLE_NAMENAME;

ENDIF;

RAISENOTICE'%ONROWSIN%.%WON'THAPPEN',

TG_OP,TG_TABLE_SCHEMA,TG_TABLE_NAMENAME;

RETURNNULL;

END;

$$LANGUAGEplpgsql;

ThesametriggerfunctioncanbeusedforboththeBEFOREandAFTERtriggers.IfyouuseitasaBEFOREtrigger,theoperationisskippedwithamessage.However,ifyouuseitasanAFTERtrigger,anERRORtriggerisraisedandthecurrent(sub)transactionisrolledback.

Itwillalsobeeasytoaddalogofthedeletedattemptsintoatableinthissametriggerfunctioninordertohelpenforcethecompanypolicy—justaddINSERTtoalogtablethatissimilartothetableinthepreviousexample.

Ofcourse,youcanmakeoneorboththemessagesmoremenacingifyouwant,byaddingsomethingsuchas”Authoritieswillbenotified!”or”Youwillbeterminated!”.

Let’stakealookathowthisworksinthefollowingcode:

postgres=#CREATETABLEdelete_test1(iint);

CREATETABLE

postgres=#INSERTINTOdelete_test1VALUES(1);

INSERT01

postgres=#CREATETRIGGERdisallow_deleteAFTERDELETEONdelete_test1FOR

EACHROWEXECUTEPROCEDUREcancel_op();

CREATETRIGGER

postgres=#DELETEFROMdelete_test1WHEREi=1;

ERROR:YOUARENOTALLOWEDTODELETEROWSINpublic.delete_test1

NoticethattheAFTERtriggerraisedanerror:

postgres=#CREATETRIGGERskip_deleteBEFOREDELETEONdelete_test1FOR

EACHROWEXECUTEPROCEDUREcancel_op();

CREATETRIGGER

postgres=#DELETEFROMdelete_test1WHEREi=1;

NOTICE:DELETEONROWSINpublic.delete_test1WON'THAPPEN

DELETE0

Thistime,theBEFOREtriggercanceledthedeleteandtheAFTERtrigger,althoughstillthere,wasnotreached.

NoteThesametriggercanalsobeusedtoenforceano-updatepolicyorevendisallowinsertstoatablethatneedstohaveimmutablecontents.

DisallowingTRUNCATEYoumayhavenoticedthattheprecedingtriggercaneasilybebypassedforDELETEifyoudeleteeverythingusingTRUNCATE.

WhileyoucannotsimplyskipTRUNCATEbyreturningNULLasopposedtotherow-levelBEFOREtriggers,youcanstillmakeitimpossiblebyraisinganerrorifTRUNCATEisattempted.CreateanAFTERtriggerusingthesamefunctionastheoneusedpreviouslyforDELETE:

CREATETRIGGERdisallow_truncate

AFTERTRUNCATEONdelete_test1

FOREACHSTATEMENT

EXECUTEPROCEDUREcancel_op();

Hereyouare,withoutTRUNCATE:

postgres=#TRUNCATEdelete_test1;

ERROR:YOUARENOTALLOWEDTOTRUNCATEROWSINpublic.delete_test1

Ofcourse,youcanalsoraisetheerrorinaBEFOREtrigger,butinthatcaseyouwillneedtowriteyourownunconditionalraise-errortriggerfunctioninsteadofcancel_op().

ModifyingtheNEWrecordAnotherformofauditingthatisfrequentlyusedistologinformationinfieldsinthesamerowasthedata.Asanexample,let’sdefineatriggerthatlogsthetimeandtheactiveuserinthelast_changed_atandlast_changed_byfieldsateachINSERTandUPDATEtrigger.Intherow-levelBEFOREtriggers,youcanmodifywhatactuallygetswrittenbychangingtheNEWrecord.Youcaneitherassignvaluestosomefieldsorevenreturnadifferentrecordwiththesamestructure.Forexample,ifyoureturnOLDfromtheUPDATEtrigger,youeffectivelymakesurethattherowcan’tbeupdated.

ThetimestampingtriggerToformthebasisofourauditlogginginthetable,westartbycreatingatriggerthatsetstheuserwhomadethelastchangeandwhenthechangeoccurred:

CREATEORREPLACEFUNCTIONchangestamp()

RETURNSTRIGGERAS$$

BEGIN

NEW.last_changed_by=SESSION_USER;

NEW.last_changed_at=CURRENT_TIMESTAMP;

RETURNNEW;

END;

$$LANGUAGEplpgsql;

Ofcourse,thisworksonlyinatablethathasthecorrectfields:

CREATETABLEmodify_test(

idserialPRIMARYKEY,

datatext,

created_bytextdefaultSESSION_USER,

created_attimestamptzdefaultCURRENT_TIMESTAMP,

last_changed_bytextdefaultSESSION_USER,

last_changed_attimestamptzdefaultCURRENT_TIMESTAMP

);

CREATETRIGGERchangestamp

BEFOREUPDATEONmodify_test

FOREACHROW

EXECUTEPROCEDUREchangestamp();

Now,let’stakealookatournewlycreatedtrigger:

postgres=#INSERTINTOmodify_test(data)VALUES('something');

INSERT01

postgres=#UPDATEmodify_testSETdata='somethingelse'WHEREid=1;

UPDATE1

postgres=#SELECT*FROMmodify_test;

-[RECORD1]---+---------------------------

id|1

data|somethingelse

created_by|postgres

created_at|2013-04-1509:28:23.966179

last_changed_by|postgres

last_changed_at|2013-04-1509:28:31.937196

TheimmutablefieldstriggerWhenyouaredependingonthefieldsintherowsaspartofyourauditrecord,youneedtoensurethatthevaluesreflectreality.Wewereabletomakesurethatthelast_changed_*fieldsalwayscontainthecorrectvalue,butwhataboutthecreated_byandcreated_atvalues?Thesecanbeeasilychangedinlaterupdates,buttheyshouldneverchange.Eveninitially,theycanbesettofalsevalues,sincethedefaultvaluescanbeeasilyoverriddenbygivinganyothervalueintheINSERTstatement.

So,let’smodifyourchangestamp()triggerfunctionintoausagestamp()function,whichmakessurethattheinitialvaluesarewhattheyshouldbeandthattheystaylikethat:

CREATEORREPLACEFUNCTIONusagestamp()

RETURNSTRIGGERAS$$

BEGIN


NEW.created_by=SESSION_USER;

NEW.created_at=CURRENT_TIMESTAMP;

ELSE

NEW.created_by=OLD.created_by;

NEW.created_at=OLD.created_at;

ENDIF;

NEW.last_changed_by=SESSION_USER;

NEW.last_changed_at=CURRENT_TIMESTAMP;

RETURNNEW;

END;

$$LANGUAGEplpgsql;

IncaseofINSERT,wesetthecreated_*fieldstotherequiredvalues,regardlessofwhattheINSERTquerytriestosetthemto.IncaseofUPDATE,wejustcarryovertheoldvalues,againoverridinganyattemptedchanges.

ThisfunctionthenneedstobeusedinordertocreateaBEFOREINSERTORUPDATEtrigger:

CREATETRIGGERusagestamp

BEFOREINSERTORUPDATEONmodify_test

FOREACHROW

EXECUTEPROCEDUREusagestamp();

Now,let’strytoupdatethecreatedauditloginformation.First,wewillneedtodroptheoriginaltriggersothatwedon’thavetwotriggersfiringonthesametable.Then,wewilltrytochangethevaluesofcreated_byandcreated_at:

postgres=#DROPTRIGGERchangestampONmodify_test;

DROPTRIGGER

postgres=#UPDATEmodify_testSETcreated_by='notpostgres',created_at=

'2000-01-01';

UPDATE1

postgres=#select*frommodify_test;

-[RECORD1]---+---------------------------

id|1

data|somethingelse

created_by|postgres

created_at|2013-04-1509:28:23.966179

last_changed_by|postgres

last_changed_at|2013-04-1509:33:25.386006

Fromtheresults,youcanseethatthecreatedinformationisstillthesame,butthelastchangedinformationhasbeenupdated.

ControllingwhenatriggeriscalledWhileitisrelativelyeasytoperformtriggeractionsconditionallyinsidethePL/pgSQLtriggerfunction,itisoftenmoreefficienttoskipinvokingthetriggeraltogether.Theperformanceeffectsoffiringatriggerarenotgenerallynoticedwhenonlyafeweventsarefired.However,ifyouarebulkloadingdataorupdatinglargeportionsofyourtable,thecumulativeeffectscancertainlybefelt.Toavoidtheoverhead,it’sbesttocallthetriggerfunctiononlywhenitisactuallyneeded.

TherearetwowaystonarrowdownwhenatriggerwillbecalledintheCREATETRIGGERcommanditself.

So,usethesamesyntaxoncemorebutwithalltheoptionsthistime:

CREATETRIGGERname

{BEFORE|AFTER|INSTEADOF}{event[ORevent…]}

[OFcolumn_name[ORcolumn_name…]]ONtable_name

[FOR[EACH]{ROW|STATEMENT}]

[WHEN(condition)]

EXECUTEPROCEDUREfunction_name(arguments)

YoucanusetheWHENclausetoonlyfireatriggerbasedonacondition(suchasacertaintimeoftheday)oronlywhencertainfieldsareupdated.Wewillnowtakealookatacoupleofexamples.

ConditionaltriggersAflexiblewaytocontroltriggersistouseagenericWHENclausethatissimilartoWHEREinSQLqueries.WithaWHENclause,youcanwriteanyexpression,exceptasubquery,thatistestedbeforethetriggerfunctioniscalled.TheexpressionmustresultinaBooleanvalue,andifthevalueisFALSE(orNULL,whichisautomaticallyconvertedtoFALSE),thetriggerfunctionisnotcalled.

Forexample,youcanusethistoenforcea“NoupdatesonFridayafternoon”policy:

CREATEORREPLACEFUNCTIONcancel_with_message()

RETURNSTRIGGERAS$$

BEGIN

RAISEEXCEPTION'%',TG_ARGV[0];

RETURNNULL;

END;

$$LANGUAGEplpgsql;

ThiscodejustraisesanexceptionwiththestringpassedasanargumentintheCREATETRIGGERstatement.NoticethatwecannotuseTG_ARGV[0]directlyasthemessagebecausethePL/pgSQLsyntaxrequiresastringconstantasthethirdelementofRAISE.

Usingtheprevioustriggerfunction,youcansetuptriggersinordertoenforcevariousconstraintsbyspecifyingboththecondition(intheWHEN(...)clause)andthemessagetoberaisedifthisconditionismetastheargumenttothetriggerfunction:

CREATETABLEnew_tasks(idSERIALPRIMARYKEY,sampleTEXT);

CREATETRIGGERno_updates_on_friday_afternoon

BEFOREINSERTORUPDATEORDELETEORTRUNCATEONnew_tasks

FOREACHSTATEMENT

WHEN(CURRENT_TIME>'12:00'ANDextract(DOWfromCURRENT_TIMESTAMP)=5)

EXECUTEPROCEDUREcancel_with_message('Sorry,wehavea"Notaskchangeon

Fridayafternoon"policy!');

Now,ifanybodytriestomodifythenew_taskstableonaFridayafternoon,theygetamessageaboutthispolicy,asshownhere:

postgres=#insertintonew_tasks(sample)values("test");

ERROR:Sorry,wehavea"NotaskchangeonFridayafternoon"policy!

NoteOnethingtonoteabouttriggerargumentsisthattheargumentlistisalwaysanarrayoftext(text[]).

AlloftheargumentsgivenintheCREATETRIGGERstatementareconvertedtostrings,andthisincludesanyNULLvalues.

ThismeansthatputtingNULLintheargumentlistresultsinthetextNULLinthecorrespondingslotinTG_ARGV.

TriggersonspecificfieldchangesAnotherwaytocontrolwhenatriggerisfiredisusingalistofcolumns.IntheUPDATEtriggers,youcanspecifyoneormorecomma-separatedcolumnstotellPostgreSQLthatthetriggerfunctionshouldonlybeexecutedifanyofthelistedcolumnschange.

ItispossibletoconstructthesameconditionalexpressionwithaWHENclause,butthelistofcolumnshascleanersyntax:

WHEN(

NEW.column1ISDISTINCTFROMOLD.column1

OR

NEW.column2ISDISTINCTFROMOLD.column2)

Acommonexampleofhowthisconditionalexpressionisusedisraisinganerroreachtimesomeonetriestochangeaprimarykeycolumn.TheISDISTINCTFROMfunctionmakessurethatthetriggerisonlyexecutedwhenthenewvalueofcolumn1isdifferentfromtheoldvalue.ThiscanbeeasilydonebydeclaringanAFTERtriggerusingthecancel_op()triggerfunction(definedpreviouslyinthischapter),asfollows:

CREATETRIGGERdisallow_pk_change

AFTERUPDATEOFidONtable_with_pk_id

FOREACHROW

EXECUTEPROCEDUREcancel_op();

postgres=#INSERTINTOnew_tasksDEFAULTVALUES;

INSERT01

packt=#SELECT*FROMnew_tasks;

id|sample

----+--------

1|

(1row)

packt=#UPDATEnew_tasksSETid=0whereid=1;

ERROR:YOUARENOTALLOWEDTOUPDATEROWSINpublic.new_tasks

VisibilitySometimes,yourtriggerfunctionsmightrunintotheMultiversionConcurrencyControl(MVCC)visibilityrulesofhowPostgreSQL’ssysteminteractswithchangestodata.

AfunctiondeclaredasSTABLEorIMMUTABLEwillneverseechangesappliedtotheunderlyingtablebytheprevioustriggers.

AVOLATILEfunctionfollowsmorecomplexruleswhichare,inanutshell,asfollows:

Thestatement-levelBEFOREtriggersdetectswhethernochangesaremadebythecurrentstatement,andthestatement-levelAFTERtriggersdetectsallofthechangesmadebythestatement.Datachangesbytheoperationtotherowcausingthetriggertofireare,ofcourse,notvisibletotheBEFOREtriggers,astheoperationhasnotoccurredyet.Changesmadebyothertriggerstootherrowsinthesamestatementarevisible,andastheorderoftherowsprocessedisundefined,youneedtobecautious.StartingfromPostgreSQL9.3,anerroristhrownifatupletobeupdatedordeletedhasalreadybeenupdatedordeletedbyaBEFOREtrigger.ThesameistruefortheINSTEADOFtriggers.Thechangesmadebythetriggersfiredinthesamecommandinthepreviousrowsarevisibletothecurrentinvocationofthetriggerfunction.Row-levelAFTERtriggersarefiredwhenallofthechangestoalltherowsoftheoutercommandarecompleteandvisibletothetriggerfunction.

Alltheserulesapplytofunctionsthatquerydatainthedatabase;theOLDandNEWrowsare,ofcourse,visibleasdescribedpreviously.

NoteThesameinformationin,perhaps,differentwordsisavailableathttp://www.postgresql.org/docs/current/static/spi-visibility.html.

http://www.postgresql.org/docs/current/static/spi-visibility.html

Mostimportantly–usetriggerscautiously!Triggersareanappropriatetoolforuseindatabase-sideactions,suchasauditing,logging,enforcingcomplexconstraints,andevenreplication(severallogicalreplicationsystemssuchasSlonyarebasedontriggersusedinproduction).However,formostapplicationlogic,itismuchbettertoavoidtriggersastheycanleadtoreallyweirdandhard-to-debugproblems.Asagoodpractice,followtherulesprovidedinthefollowingtable:

Rule Description

Rule1 Donotchangedataintheprimarykey,foreignkey,oruniquekeycolumnsofanytable

Rule2 Donotupdaterecordsinthetablethatyoureadduringthesametransaction

Rule3 Donotaggregateoverthetableyouareupdating

Rule4 Donotreaddatafromatablethatisupdatedduringthesametransaction

VariablespassedtothePL/pgSQLTRIGGERfunctionThefollowingisacompletelistofthevariablesavailabletoatriggerfunctionwritteninPL/pgSQL:

OLD,NEW RECORD

Thisrecordsthebeforeandafterimagesoftherowonwhichthetriggeriscalled.OLDisunassignedforINSERTandNEWisunassignedforDELETE.

BothareUNASSIGNEDinstatement-leveltriggers.

TG_NAME name Thisdenotesthenameofthetrigger(thisandfollowingfromthetriggerdefinition).

TG_WHEN text BEFORE,AFTER,orINSTEADOFarethepossiblevaluesofthevariable.

TG_LEVEL text ROWorSTATEMENTarethepossiblevaluesofthevariable.

TG_OP text INSERT,UPDATE,DELETE,orTRUNCATEarethepossiblevaluesofthevariable..

TG_RELID oid ThisdenotestheOIDofthetableonwhichthetriggeriscreated.

TG_TABLE_NAME nameThisdenotesthenameofthetable(theoldspellingTG_RELNAMEisdeprecatedbutstillavailable).

TG_TABLE_SCHEMA name Thisdenotestheschemanameofthetable.

TG_NARGS,TG_ARGV[]

Int,text[]

Thisdenotesthenumberofargumentsandthearrayoftheargumentsfromthetriggerdefinition.

TG_TAG textThisisusedinDDLoreventtriggers.Thisvariablecontainsthenameofthecommandthatresultedinthetriggerinvocation.Moreinformationonthisinthenextchapter.

Youcanreadmoreaboutthevariablesathttp://www.postgresql.org/docs/current/static/plpgsql-trigger.html.

http://www.postgresql.org/docs/current/static/plpgsql-trigger.html

SummaryAtriggerisabindingofasetofactionstocertainoperationsperformedonatableorview.Thissetofactionsisdefinedinaspecialtriggerfunctionwhichisdistinguishedbyspecifyingthetypeofthereturnedvaluetobeofaspecialpseudotypetrigger.So,eachtimeanoperation(INSERT,UPDATE,DELETE,orTRUNCATE)isperformedonthetable,thistriggerfunctioniscalledbythesystem.

Itcanbeexecutedeitherforeachroworforeachstatement.Ifexecutedforeachrow(row-leveltrigger),thefunctionispassedspecialvariablessuchasOLDandNEW.

Thiswillcontaintherow’scontents,asitiscurrentlyinthedatabase(OLD)andasitisthemomentthetriggerfunctioniscalled(NEW).WheretheOLDorNEWvalueismissing,itispassedasundefined.Ifexecutedonceperstatement(thestatement-leveltrigger),bothOLDandNEWareunassignedforalltheoperations.

Thetriggerfunctionforrow-leveltriggersonINSERT,UPDATE,andDELETEcanbesettoexecuteeitherBEFOREorAFTERtheoperationonatableandcanbesettoexecutetheINSTEADOFoperationonview.

Thetriggerfunctionforstatement-leveltriggersonINSERT,UPDATE,andDELETEcanbesettoexecuteeitherBEFOREorAFTERtheoperationonbothtablesandviews.

WhileTRUNCATEislogicallyaspecial,non-MVCCformofa“deleteall”statement,noONDELETEtriggerswillfireinthecaseofTRUNCATE.Instead,youcanuseaspecialONTRUNCATEtriggeronthesametable.Onlystatement-levelonTRUNCATEtriggersarepossible.WhileyoucannotskipstatementtriggersbyreturningNULL,youcanraiseanexceptionandabortthetransaction.

ItisalsonotpossibletodefineanyONTRUNCATEtriggersonviews.

Inthenextchapter,wewilltakealookatthenewPostgreSQLeventtriggerfeature.EventtriggersallowyoutoexecutetriggerswhenaDataDefinitionLanguage(DDL)statementisexecutedonatable.

Chapter6.PostgreSQLEventTriggersPostgreSQL9.3introducedaspecialtypeoftriggerstocomplementthetriggermechanismwediscussedintheprecedingchapter.Thesetriggers,aredatabase-specificandarenotattachedtoaspecifictable.Unlikeregulartriggers,eventtriggerscaptureDDLevents.EventtriggerscanbetheBEFOREandAFTERtriggers,andthetriggerfunctioncanbewritteninanylanguageexceptSQL.OtherpopulardatabasevendorssuchasOracleandSQLServeralsoprovideasimilarfunctionality.

OnecanthinkofseveralusecasesforDDLtriggers.ThemostpopularoneamongtheDBAsnormallyistodoanaudittrail.You,asaDBA,mightwanttoaudittheusersandtheDDLcommands.Schemachanges,inaproductionenvironment,shouldbedoneverycarefully;hence,theneedforanaudittrailisapparent.Eventtriggersaredisabledinthesingleusermodeandcanonlybecreatedbyasuperuser.


UsecasesforeventtriggersAfullaudittrailexampleusingPL/pgsqlPreventingschemachangesusingeventtriggers

UsecasesforcreatingeventtriggersInadditiontoanaudittrail,thefollowingcanbevalidusecasesforaneventtrigger:

ReplicatingDDLchangestoaremotelocationCascadingaDDLPreventingchangestotables,exceptduringapredefinedwindowProvidinglimitedDDLcapabilitytodevelopers/supportstaffusingsecuritydefinerfunctionsDisablingcertainDDLcommandsbasedonacriteriaPerformanceanalysistoseehowlongacommandtakesbetweenddl_command_startandddl_command_end

NotethattheeventtriggersupportforPL/pgsqlisnotyetcompletein9.3.ThereareseveralfeaturesthatarebeingworkeduponandwillbeavailableinfuturePostgreSQLversions,hopefully.Theseareafewofthemostnotablefeaturesthataremissingsofar:

There’snoinformationabouttheobjectthataDDLtargetsThere’snoaccesstothecommandstringThere’snosupportforthegeneratedcommands

AcommandsuchasCREATETABLEfoo(idserialPRIMARYKEY)willalsoresultintheCREATESEQUENCEandCREATEINDEXcommandstobeexecutedinternally.However,thecurrentDDLtriggerswillnotbeabletologthesegeneratedcommands.

CreatingeventtriggersEventtriggersarecreatedusingtheCREATEEVENTTRIGGERcommand.Beforeyoucancreateaneventtrigger,youneedafunctionthatthetriggerwillexecute.ThisfunctionmustreturnaspecialtypecalledEVENT_TRIGGER.Ifyouhappentodefinemultipleeventtriggers,theyareexecutedinthealphabeticalorderoftheirnames.

Currently,eventtriggersaresupportedonthreeevents,asfollows:

ddl_command_start:ThiseventoccursjustbeforeaCREATE,ALTER,orDROPDDLcommandisexecutedddl_command_end:ThiseventoccursjustafteraCREATE,ALTER,orDROPcommandhasfinishedexecutingsql_drop:Thiseventoccursjustbeforetheddl_command_endeventforthecommandsthatdropdatabaseobjects

YoucanspecifyaWHENclausewiththeCREATEEVENTTRIGGERcommand,sothattheeventisfiredonlyforthespecifiedcommands.

TheeventtriggerPL/pgSQLfunctionshaveaccesstothefollowingnewvariablesintroducedin9.3:

TG_TAG:Thisvariablecontainsthe“tag”orthecommandforwhichthetriggerisexecuted.Thisvariabledoesnotcontainthefullcommandstring,butjustatagsuchasCREATETABLE,DROPTABLE,ALTERTABLE,andsoon.TG_EVENT:Thisvariablecontainstheeventname,whichcanbeddl_command_start,ddl_comman_end,andsql_drop.

NoteAcompletematrixoftheeventtriggerfiringmechanismcanbefoundinthePostgreSQLdocumentationathttp://www.postgresql.org/docs/current/static/event-trigger-matrix.html.

http://www.postgresql.org/docs/current/static/event-trigger-matrix.html

CreatinganaudittrailLet’snowtakealookatthecompleteexampleofaneventtriggerthatcreatesanaudittrailofsomeDDLcommandsinthedatabase:

CREATETABLEtrack_ddl

(

eventtext,

commandtext,

ddl_timetimestamptz,

usrtext

);

CREATEORREPLACEFUNCTIONtrack_ddl_function()

RETURNSevent_trigger

AS

$$

BEGIN

INSERTINTOtrack_ddlvalues(tg_tag,tg_event,now(),session_user);

RAISENOTICE'DDLlogged';

END


CREATEEVENTTRIGGERtrack_ddl_eventONddl_command_start

WHENTAGIN('CREATETABLE','DROPTABLE','ALTERTABLE')

EXECUTEPROCEDUREtrack_ddl_function();

CREATETABLEevent_check(iint);

SELECT*FROMtrack_ddl;

-[RECORD1]------------------------

event|CREATETABLE

command|ddl_command_start

ddl_time|2014-04-1316:58:40.331385

usr|testusr

Theexampleisactuallyquitesimple.Here’swhatwehavedoneintheexample:

1. First,wecreatedatablewherewestoretheauditlog.Thetableisquitesimpleatthemoment,duetoalimitedamountofinformationthatiscurrentlyavailabletoaPL/pgSQLfunction.Westorethetag,theevent,thetimestampwhenthistriggerisexecuted,andtheuserwhoexecutedthecommand.

2. Wethencreateafunctionthatisexecutedbythetriggerwheneveritisfired.Thefunctionissimpleenoughfornow.ItmustreturnthetypeEVENT_TRIGGER.ItlogstheDDLinformationintheaudittrailtable.ThefunctioncreatedisSECURITYDEFINER.Thereasonwhythisisdoneisbecauseotherusersinthedatabasedon’thaveanyprivilegesontheaudittrailtableandwedon’tactuallywantthemtoknowitisthere.Thisfunctionisexecutedasthedefiner(whichisasuperuser),andtheuseofsession_userensuresthatwelogtheuserwhologgedintothedatabase,andnottheonewhoseprivilegesareusedtoexecutethefunction.

3. Wethencreateaneventtriggerthatonlyexecuteswhencertaincommandssuchas

CREATE,DROP,orALTERTABLEareexecutedbyauser.

Wethencreateanexampletableandnotethatthecommandisindeedloggedintheaudittrailtable.

PreventingschemachangesAnothercommonusecaseforeventtriggersistopreventtheexecutionofcertaincommandsbasedonaspecificcondition.Ifyouonlywanttostopusersfromexecutingsomecommands,youcanalwaysrevoketheprivilegesusingmoreconventionalmeans.Thetriggersmaycomeinhandyifyouwanttodothisbasedonacertaincondition,let’ssay,timeoftheday:

CREATEORREPLACEFUNCTIONabort_create_table_func()

RETURNSevent_trigger

AS

$$

DECLARE

current_hourint:=extract(hourfromnow());

BEGIN

ifcurrent_hourBETWEEN9AND16

then

RAISEEXCEPTION'Notasuitabletimetocreatetables';

endif;

END;


CREATEEVENTTRIGGERabort_create_tableONddl_command_start

WHENTAGIN('CREATETABLE')

EXECUTEPROCEDUREabort_create_table_func();

Theprecedingcodeis,again,quitesimple:

1. First,wecreateatriggerfunctionthatpreventsachangeifthecurrenthourofthedayisbetween9to16.

2. WecreateatriggerthatexecutesthisfunctiononlywhenaCREATETABLEcommandisexecuted.

Youcanextendthisexampletoincludemorecomplexconditionsandalsobasethedecisiononthecurrentstateofthedatabase.ThiskindofflexibilityisnotavailablewhenyouusesimpleprivilegemanagementbasedonGRANTandREVOKE.

AroadmapofeventtriggersAsyoucansee,thecurrentimplementationofeventtriggersinPL/pgsqlandPostgreSQL9.4isratherlimited.However,therearemorechangesandfeaturesplannedfortheupcomingreleasesthatwillexpandthescope,inwhicheventtriggerswillbecomemoreuseful.Herearethehighlightsofwhatwecanlookforwardtointhefuture:

Accesstomoreinformation:MoreTG_*variablesaregoingtobeavailable,inordertoprovidemoreinformationaboutthecommandthatisrunningandonwhichobjectitisrunning.WecanlookforwardtothefollowingvariablesinfuturePostgreSQLversions:

TG_OBJECTID

TG_OBJECTNAME

TG_SCHEMANAME

TG_OPERATION

TG_OBTYPENAME

TG_CONTEXT

Accessors:Thesearejustfunctionswhichgiveyousomeinformation.Inthiscasethepg_get_event_command_stringfunctionwillgivethefullcommandstringoftheDDLcommandthatcausedtheeventtriggertofire.

pg_get_event_command_string()

SupportofDROPCASCADE:Here,eventtriggerswillbefiredforeachobjecteffectedinaDROPCASCADEcall.INSTEADOF:Theideahere,isforaneventtriggertotakecontrolofacommand,analogoustotheINSTEADOFDMLtriggers.CREATETABLEonINSERT:Theideahere,istojustcreateanewtablewheneverwereceiveaninsertforthefirsttimeandthetabledoesn’texist.

NoteIfyouwanttolearnmoreaboutthefeaturesinprogressandthepatches,whicharebeingdiscussedrelatedtoeventtriggersupportinPostgreSQL,pleasefollowthewikipageathttps://wiki.postgresql.org/wiki/Event_Triggers.

https://wiki.postgresql.org/wiki/Event_Triggers

SummaryEventtriggersarenewinPostgreSQL9.3,andthecommunityisstillworkingonvariousfeaturesinordertomakethemmoreuseful.Youcanusethesetriggersforvariouspurposes,includingauditingDDLcommandsandmakinglocalcustomizedpoliciesregardingtheexecutionofDDLcommandsinthedatabase.Eventtriggersaredisabledinsingleusermode.

EventtriggersarecreatedusingtheCREATEEVENTTRIGGERcommand,andtheyexecuteafunctionthatreturnsaspecialtypecalledEVENT_TRIGGER.Therearethreetypesofeventsthatarecurrentlysupported:ddl_command_start,ddl_command_end,andsql_drop.YoucanlimitatriggerexecutionbyatagusingaWHENclause,ifyouwanttofireitforspecificcommandsonly.

Currently,therearetwonewvariablesavailableintheeventtriggerfunctioncalledTG_TAGandTG_EVENTthatprovideinformationaboutthetagandtheeventofthetrigger.FuturereleasesofPostgreSQLwillexposemorevariablesthatmakeitpossibletoauditcompleteinformationaboutfiringDDL,includingobjectsandthecommandstring.

Inthenextchapter,wewilldiscussthedifferencesbetweenrestrictedandunrestrictedlanguagesandwewilltakealookatthepracticalexamplesofboth.

Chapter7.DebuggingPL/pgSQLThischapterisentirelyoptional.Sinceyouhaveonlyproducedthehighestqualitybug-freecodeusingthebestpossiblealgorithms,thistextisprobablyawasteofyourtime.Ofcourse,yourfunctionsparseperfectlyonthefirsttry.Yourviewsshowexactlywhattheyshould—accordingtotheenviouslycompletebusinessandtechnicaldocumentationthatyouwrotelastmonth.Thereisnoneedforversioncontrolonyourprocedures,astherehasonlyeverbeenaVersion1.

Sinceyouarestillreadingthis,I’msurethatyou’reawholelotmorelikeme.Ispendabout10percentofmytimewritingnewcodeandabout90percentofiteditingthemistakesandoversightsthatI(andothers)madeinthefirst10percent.Infact,itcanbearguedthatnonewcodeiseverwrittenatall.Actually,amoreaccuratedescriptionoftheprocessisthatadumbassertionismadeandthenitisediteduntilthecustomercannolongerstandtheQualityAssurance(QA)process.Wethenshiptheresultinthehopesofbeingusefultotheenduser.Wasthattoomuchofrealityforyou?Sorry.

Theobjectiveofthischapteristomakeyoufasteratmakingmistakes.Asaby-product,youwillalsolearnhowtodiagnoseandfixthematanalarmingrate.Theneteffectofthis,wearehoping,isthatyourbosswillassumeyouwroteitcorrectlythefirsttime.Thisis,ofcourse,aliebutaveryusefulone.

Thisconceptiscriticaltoagilesoftwaredevelopment.Inthisphilosophy,itiscalled“prototyping.”Theideaistocreateafeaturequicklyanddemonstrateitasaconversationpoint,ratherthantryingtoproduceanentiresystem(presumablyperfectly)fromconceptualdocumentation.Otherauthorsrefertoitas“failingquickly.”Itrecognizesthefactthatthefirstthreeorfourdevelopmentiterationswillprobablynotbeacceptabletothecustomerandshouldn’tbeadvertisedasfinaluntilsomediscussionhastakenplace.

Thisprocesseffectivelyrequiresthedeveloperto“live”inthedebugger.Thedevelopercontinuallychangestheoutputsandroutinesuntilthedesiredeffectisachieved.PostgreSQLhasawonderfulsetofdebuggingtoolsavailabletohelpyoufixyourmess.Letmeshowyouhowtheywork.

ManualdebuggingwithRAISENOTICEHere’sthefirstpromisedexample:

CREATEORREPLACEFUNCTIONformat_us_full_name_debug(

prefixtext,

firstnametext,

mitext,

lastnametext,

suffixtext)

RETURNStextAS

$BODY$

DECLARE

fname_mitext;

fmi_lnametext;

prefix_fmiltext;

pfmil_suffixtext;

BEGIN

fname_mi:=CONCAT_WS('',CASEtrim(firstname)WHEN''THENNULLELSE

firstnameEND,CASEtrim(mi)WHEN''THENNULLELSEmiEND||'.');

RAISENOTICE'firstnamemi.:%',fname_mi;

fmi_lname:=CONCAT_WS('',CASEfname_miWHEN''THENNULLELSEfname_mi

END,CASEtrim(lastname)WHEN''THENNULLELSElastnameEND);

RAISENOTICE'firstnamemi.lastname:%',fmi_lname;

prefix_fmil:=CONCAT_WS('.',CASEtrim(prefix)WHEN''THENNULLELSE

prefixEND,CASEfmi_lnameWHEN''THENNULLELSEfmi_lnameEND);

RAISENOTICE'prefix.firstnamemilastname:%',prefix_fmil;

pfmil_suffix:=CONCAT_WS(',',CASEprefix_fmilWHEN''THENNULLELSE

prefix_fmilEND,CASEtrim(suffix)WHEN''THENNULLELSEsuffix||'.'

END);

RAISENOTICE'prefix.firstnamemilastname,suffix.:%',pfmil_suffix;

RETURNpfmil_suffix;

END;

$BODY$

LANGUAGEplpgsqlVOLATILE;

Inthisexample,weformataperson’sfullnameusingthemagicoftheNULLpropagation.

TheNULLpropagationiswhatoccurswhenanyorallofthemembersofanexpressionarenull.Inthemyvar:=null||'something'expression,myvarwillevaluatetonull.PostgreSQL9.1introducedahandynewfunctionnamedCONCAT_WS(concatenatewithaseparator)totakeadvantageofthiseffect.

Takeanexampleofthefollowinglineofcode:

lastfirst:=CONCAT_WS(',',lastname,firstname);

Theprecedingcodewillnotprintthecommaandwhitespacebetweenlastnameandfirstnameifeitherfirstnameorlastnameisnotpresent.Thiseffectisusedintheformat_us_address()functionwithmanylevelsofnestinginordertoprovideanaddressthatisvisuallyappealingaswellaspostal-processingfriendly.

ThereareseveralstatementsinthecodeexamplethatshowyouhowtouseRAISENOTICE

alongwithsometextandavariabletoprovidedebugginginformationasthefunctionisbeingcalled.Forexample,runningourfunctioninpgAdmin3willproducesomenotificationmessages,asshownhere:

NOTICE:firstnamemi.:KirkL.

NOTICE:firstnamemi.lastname:KirkL.Roybal

NOTICE:prefix.firstnamemilastname:Mr.KirkL.Roybal

NOTICE:prefix.firstnamemilastname,suffix.:Mr.KirkL.Roybal,

Author.

YoucanseethesemessagesinpgAdmin3undertheMessagestab,asshowninthefollowingscreenshot:

Theoutputofthesamequeryinthecommand-linepsqlclientisshowninthefollowingcode:

kroybal=#SELECT

format_us_full_name_debug('Mr','Kirk','L','Roybal','Author');

NOTICE:firstnamemi.:KirkL.

NOTICE:firstnamemi.lastname:KirkL.Roybal

NOTICE:prefix.firstnamemilastname:Mr.KirkL.Roybal

NOTICE:prefix.firstnamemilastname,suffix.:Mr.KirkL.Roybal,

Author.

format_us_full_name_debug

-----------------------------

Mr.KirkL.Roybal,Author.

(1row)

Ifyoudon’tseetheRAISENOTICEmessagesonyourscreen,youshouldcheckclient_min_messagesinyoursession,asshowninthefollowingcode.ItshouldbeNOTICEorhighertoseethemessages:

kroybal=#SHOWclient_min_messages;

client_min_messages

---------------------

notice

(1row)

ThrowingexceptionsTheRAISEcommandtakesseveraloperatorsexceptNOTICE.Thecommandwillalsothrowexceptionsthatareintendedforthecallingcodetocatch.Thefollowingisanexampleofhowtocreateanexception:

CREATEORREPLACEFUNCTIONvalidate_us_zip(zipcodeTEXT)

RETURNSboolean

AS$$

DECLARE

digitstext;

BEGIN

—removeanythingthatisnotadigit(POSIXcompliantly,please)

digits:=(SELECTregexp_replace(zipcode,'[^[:digit:]]','','g'));

IFdigits=''THEN

RAISEEXCEPTION'Zipcodedoesnotcontainanydigits-->%',digits

USINGHINT='IsthisaUSzipcode?',ERRCODE='P9999';

ELSIFlength(digits)<5THEN

RAISEEXCEPTION'Zipcodedoesnotcontainenoughdigits-->%',digits

USINGHINT='Zipcodehaslessthan5digits.',ERRCODE='P9998';

ELSIFlength(digits)>9THEN

RAISEEXCEPTION'Unnecessarydigitsinzipcode-->%',digitsUSING

HINT='Zipcodeismorethan9digits.',ERRCODE='P9997';

ELSIFlength(digits)>5ANDlength(digits)<9THEN

RAISEEXCEPTION'Zipcodecannotbeprocessed-->%',digitsUSINGHINT

='Zipcodeabnormallength.',ERRCODE='P9996';

ELSE

RETURNtrue;

ENDIF;

END;

$$LANGUAGEplpgsql;

ThedeveloperdefinestheERRCODEvalues.Inthisexample,IusedthegeneralPL/pgSQLerrorcodevalue(P0001orplpgsql_error),startedatthetopoftherange(P9999)oferrors,anddecrementedforeachtypeoferrorthatIwishedtoexpose.ThisisaverysimplistictechniquedesignedtopreventoverlapinthefuturefromerrorcodesusedbyPL/pgSQL.Youarefreetoinventanyerrorcodesyouwant,butyouwouldbewelladvisedtoavoidthosealreadylistedinthedocumentationathttp://www.postgresql.org/docs/current/static/errcodes-appendix.html.

Thefollowingcodeisusedtocaptureanyerrorsthrowninthepreviousexample:

CREATEORREPLACEFUNCTIONget_us_zip_validation_status(zipcodetext)

returnstext

AS

$$

BEGIN

SELECTvalidate_us_zip(zipcode);

RETURN'PassedValidation';

EXCEPTION

WHENSQLSTATE'P9999'THENRETURN'Non-USZipCode';

WHENSQLSTATE'P9998'THENRETURN'Notenoughdigits.';

WHENSQLSTATE'P9997'THENRETURN'Toomanydigits.';

http://www.postgresql.org/docs/current/static/errcodes-appendix.html

WHENSQLSTATE'P9996'THENRETURN'Between6and8digits.';

RAISE;—SomeotherSQLerror.

END;

$$

LANGUAGE'plpgsql';

Thiscodecanbecalledasfollows:

SELECTget_us_zip_validation_status('34955');

get_us_zip_validation_status

------------------------------

PassedValidation

(1row)

root=#SELECTget_us_zip_validation_status('349587');


------------------------------

Between6and8digits.

(1row)



------------------------------

Toomanydigits.

(1row)

root=#SELECTget_us_zip_validation_status('BNHCGR');


------------------------------

Non-USZipCode

(1row)



------------------------------

Notenoughdigits.

(1row)

LoggingtoafileTheRAISEstatementexpressioncanbesenttoalogfileusinglog_min_messages.Thisparameterissetinpostgresql.conf.Thevalidvaluesaredebug5,debug4,debug3,debug2,debug1,info,notice,warning,error,log,fatal,andpanic.

Thedefaultlogginglevelisdependentonthepackagingsystem.OnUbuntu,thedefaultlogginglevelisinfo.ThelogginglevelscorrespondtothesameexpressionsfortheRAISEstatement.Asadeveloper,youcanraiseanyofthemessagesthatareavailableandhavethemrecordedinthelogfileforanalysislater.

ThesimplestwaytopostamessagetothePostgreSQLdaemonlogfileisusingRAISELOG:

RAISELOG'WhyamIdoingthis?';

Thislogfileisusuallylocatedinthe<data_directory>/pg_logfolder.

TheadvantagesofRAISENOTICEUsingtheRAISENOTICEformofdebugginghasseveraladvantages.Itcanbeusedeasilyandrepeatedlywithscriptsforregressiontesting.Thisisveryeasilyaccomplishedwiththecommand-lineclient.Considerthefollowingstatement:

psql-qtc"SELECTformat_us_full_name_debug

('Mr','Kirk','L.','Roybal','Author');"

Theprecedingstatementproducesthefollowingoutputtostdout:

NOTICE:firstnamemi.:KirkL..

NOTICE:firstnamemi.lastname:KirkL..Roybal

NOTICE:prefix.firstnamemilastname:Mr.KirkL..Roybal

NOTICE:prefix.firstnamemilastname,suffix.:Mr.KirkL..Roybal,

Author.

Mr.KirkL..Roybal,Author.

Becauseaconstantsetofinputparametersshouldalwaysproduceaknownoutput,itisveryeasytousecommand-linetoolsinordertotestforexpectedoutputs.Whenyouarereadytodeployyournewlymodifiedcodetotheproductionsystem,runyourcommand-lineteststoverifythatallofyourfunctionsstillworkasexpected.

RAISENOTICEisincludedwiththeproductandrequiresnoinstallation.ItsadvantagewillbecomeclearerlaterinthechapterwheretheratherpainfulinstallationprocedureforthePL/pgSQLdebuggerisexplained.

TheRAISEstatementiseasytounderstand.Thesyntaxisstraightforward,anditiswelldocumentedathttp://www.postgresql.org/docs/current/static/plpgsql-errors-and-messages.html.

RAISEworksinanydevelopmentenvironmentandhasbeenaroundforaverylongtime,inalmosteveryversionofPostgreSQLoneveryoperatingsystem.IhaveuseditwithpgAdmin3,phpPgAdmin,aswellasthecommand-linetoolpsql.

http://www.postgresql.org/docs/current/static/plpgsql-errors-and-messages.html

Theseattributes,takentogether,makeRAISEaveryattractivetoolforsmall-scaledebugging.

ThedisadvantagesofRAISENOTICEUnfortunately,therearesomedisadvantagestousingthismethodofdebugging.TheprimarydisadvantageisthatyouneedtoremovetheRAISEstatementswhentheyarenolongernecessary.Themessagestendtoclutterupthepsqlcommand-lineclientandaregenerallyannoyingtootherdevelopers.Thelogmayfillupquicklywithuselessmessagesfrompreviousdebuggingsessions.TheRAISEstatementsneedtobewritten,commentedout,andrestoredwhenneeded.Theymaynotcovertheactualbugbeingsought.Theyalsoslowdowntheexecutionoftheroutine.

YouwillalsofindChapter13,PublishingYourCodeasPostgreSQL,quiteinformative.Thischapterincludessomeexamples(andanextremelyhandywaytoinstallthem)thatwillbeusefulhereinthispartofthebook.Theexampleswillbeshowninthetextofthischapteraswell,buttheywillbequiteabiteasierforyoutoinstallasanextension.

VisualdebuggingThePL/pgSQLdebuggerisaprojecthostedonPostgreSQLGitthatprovidesadebugginginterfaceintoPostgreSQLVersion8.2orhigher.Theprojectishostedathttp://git.postgresql.org/gitweb/?p=pldebugger.git;a=summary.

ThePL/pgSQLdebuggerletsyoustepthroughthePL/pgSQLcode,setandclearbreakpoints,viewandmodifyvariables,andwalkthroughthecallstack.

Asyoucanseefromthedescription,thePL/pgSQLdebuggercanbequiteahandylittletooltohaveinyourarsenal.

http://git.postgresql.org/gitweb/?p=pldebugger.git;a=summary

InstallingthedebuggerOK,nowwewillmovepasttheglamourandactuallygetthedebuggerrunningonoursystem.IfyouinstallPostgreSQLwithoneofthepackagesthatcontainsthedebugger,theinstallationisprettysimple.Otherwise,youwillneedtobuilditfromthesource.

AdetaileddiscussionofhowtoinstallthePL/pgSQLdebuggerfromthesourceisbeyondthescopeofthisbook,butIwilljustlistthesetofstepstoinstallthedebuggerquickly.ThebestwaytobuildthesourcewillbetopullthelatestversionfromtheGitrepositoryandfollowtheREADMEfileinthedirectory.IfyouwanttogetstartedwithitquicklyandyouhaveaWindowsmachineavailable,thesimplestwaytousethedebuggerisusingthePostgreSQLWindowsinstallerfromhttp://www.postgresql.org/download/windows/.

http://www.postgresql.org/download/windows/

InstallingthedebuggerfromthesourceHereisasetofsimplestepsthatshouldgetyouupandrunningifyouwanttoinstallfromthesource:

1. ClonetheGitrepositoryasshown.Youcanviewtherepositoryathttp://git.postgresql.org/gitweb/?p=pldebugger.git;a=summary:

gitclonehttp://git.postgresql.org/git/pldebugger.git

2. Copythispldebugger/directorytocontrib/inyourPostgreSQLsourcetree.3. Runmake&&makeinstallinthepldebuggerfolder.4. Modifytheshared_preload_librariesconfigurationoptioninpostgresql.confas

follows:

shared_preload_libraries='$libdir/plugin_debugger'

5. RestartPostgreSQL.6. Createthedebuggerextension:

CREATEEXTENSIONpldbgapi;

ThisshouldinstallthePLdebuggerandyoushouldbeabletouseitwithpgAdmin3.

TipYoucanalsoinstallPostgreSQLusingEnterpriseDB’sone-clickinstallersformostplatforms,whichalsoincludesthePLdebuggerathttp://www.enterprisedb.com/products-services-training/pgdownload.

http://git.postgresql.org/gitweb/?p=pldebugger.git;a=summary

http://www.enterprisedb.com/products-services-training/pgdownload

InstallingpgAdmin3ThePL/pgSQLdebuggermoduleworkswithpgAdmin3.Youdon’tneedtoperformspecialstepswiththeinstallationofpgAdmin3forthedebuggertofunction.Installitasusualfromyourpackagemanagerontheplatformthatyouareusing.ForUbuntu10.04LTS,thecommandisasfollows:

sudoapt-getinstallpgadmin3

UsingthedebuggerWhenthedebuggerisavailableforaparticulardatabase,itcanbeseeninthecontextmenuwhenyouright-clickonaPL/pgSQLfunction.Wehavealreadycreatedsomeofthedebuggersintheearlierpartofthischapter.Usingformat_us_full_nameasanexample,right-clickonitandnavigatetoDebugging|Debug:

Asaresult,youwillseethefollowingdialog:

Entersomevaluesintothecolumns,asseenintheprecedingscreenshot,andclickontheDebugbutton.Youwillbedepositedintothedebugger:

Thiswillallowyoutostepthroughthecodeandseethevaluesofanyvariablesastheyarebeingchanged.Clickonthestep-intobuttonafewtimestoseehowthevaluesaremodifiedasthefunctionisperformed.

TheadvantagesofthedebuggerThePL/pgSQLdebuggerdoesnotrequireanyresourcesontheserverwhennotactuallyinuse.BecauseitisinvokedmanuallyfromwithinpgAdmin3,itisnotresidentinmemoryuntilitisactuallycalledupon.Thisarchitecturedoesnotrequireanybackgroundprocessesoradditionaldaemonsforthesakeofdebugging.

Also,thePL/pgSQLdebuggerdoesnotrequireanyspecial“calling”functionstobewritteninordertoinvokethedebuggingprocess.Therearenoerrorstotrapandnotablesoferrorcodestointerpret.Everythingnecessarytothedebuggingprocessisavailableinasimplewindow.

Ifyouconnecttoyourdatabaseasasuperuser,youalsohavetheabilitytosetaglobalbreakpoint.Thisbreakpointcanbesetonanyfunctionortriggeranditwillstopthenexttimeanycodepathcallsthefunction.Thisisparticularlyusefulifyouwanttodebugyourfunctionsortriggersinthecontextofyourentirerunningapplication.

ThegreatestadvantageofthePL/pgSQLdebuggeristhatitdoesnotrequireanyspecialrigginginthefunctionsthatarebeingdebugged.

Thereisnocodetobeinsertedorremoved,andgoodcodingpracticesdon’tneedtobemodifiedwithrespecttodebugging.Thereisnopossibilityto“forget”thedebuggingcodewhenmovingtoproduction.AllofyourPL/pgSQLfunctionsarenowinstantlyreadytodebugwithoutanyspecialaction.

Thedisadvantagesofthedebugger

Asyouhavebecomepainfullyaware,theinstallationofthedebuggerleavesalottobedesired.ThisdebuggerhasnotbecomeverypopularinthePostgreSQLcommunityatlargebecauseoftheratherlargelearningcurveinvolved,andthat’sjusttogetitinstalled.

Thisformofdebuggingismeantforpersonalproductivitywhileactivelydevelopingfunctions.Itdoesnotworkwellasanautomationtool.

SummaryThedebuggingmethodsthatwehaveseeninthischapteraredesignedtobeusedinconjunctionwithoneanother.Theycomplementeachotheratdifferentpointsinthedevelopmentprocess.WheredebuggingusingthePL/pgSQLdebuggerishighlyeffectivewhileeditinganexisting(hopefullywell-written)function,otherformsofdebuggingmaybebettersuitedtothequalityassuranceorautomateddataprocessingapplications.

BecausethePL/pgSQLdebuggerismeanttobeavisualtooltoworkwithinpgAdmin3,itispossiblethatthedevelopermaywanttoforegothevisualdebuggerintheinterestofsomeotherfeature.

Inthenextchapter,wewilltakealookathowtowritesomeadvancedfunctionsinC.

Chapter8.UsingUnrestrictedLanguagesYoumayhavenoticed,thatsomeofthePLsinPostgreSQLcanbedeclaredasuntrusted.Theyallendintheletterutoremindyouthattheyareuntrustedeachtimeyouusethemtocreateafunction.Unrestrictedlanguagesallowyoutodothingsthatrestrictedortrustedlanguagesarenotallowedtodo;forexample,interactingwiththeenvironmentandcreatingfilesandopeningsockets.Inthischapter,wewilllookatsomeexamplesindetail.

Thisuntrustednessbringsupmanyquestions:

Doesbeinguntrustedmeanthatsuchlanguagesaresomehowinferiortotrustedones?CanIstillwriteanimportantfunctioninanuntrustedlanguage?Willtheysilentlyeatmydataandcorruptthedatabase?

Theanswersareno,yes,andmayberespectively.Let’snowdiscussthesequestionsinorder.

Areuntrustedlanguagesinferiortotrustedones?No,onthecontrary,theselanguagesareuntrustedinthesamewaythatasharpknifeisuntrustedandshouldbekeptoutofthereachofverysmallchildren,unlessthereisadultsupervision.TheyhaveextrapowersthatordinarySQL,oreventhetrustedlanguages(suchasPL/pgSQL)andtrustedvariantsofthesamelanguage(PL/PerlversusPL/PerlU)don’thave.

Youcanusetheuntrustedlanguagestodirectlyreadandwriteontheserver’sdisks,andyoucanuseittoopensocketsandmakeInternetqueriestotheoutsideworld.Youcanevensendarbitrarysignalstoanyprocessrunningonthedatabasehost.Generally,youcandoanythingthenativelanguageofthePLcando.

However,youprobablyshouldnottrustarbitrarydatabaseuserstohavetherighttodefinefunctionsintheselanguages.Alwaysthinktwicebeforegivingallprivilegesonanuntrustedlanguagetoauserorgroup,byusingthe*ulanguagesforimportantfunctions.

Canyouuseuntrustedlanguagesforimportantfunctions?Absolutely!Sometimes,itmaybetheonlywaytoaccomplishsometasksfrominsidetheserver.Performingsimplequeriesandcomputationsshoulddonothingharmfultoyourdatabase,andneithershouldconnectingtotheexternalworldforsendinge-mails,fetchingwebpages,orperformingSOAPrequests.However,becarefulaboutperformingoperationsthatmaycausedelaysandevenqueriesthatgetstuck,butthesecanusuallybedealtwithbysettinganupperlimitastohowlongaquerycanrun,byusinganappropriatestatementtime-outvalue.Settingareasonablestatementtime-outvaluebydefaultisagoodpracticeanyway.

So,ifyoudon’tdeliberatelydoriskythings,theprobabilityofharmingthedatabaseisnobiggerthanusinga“trusted”(alsoknownasrestricted)variantofthelanguage.However,ifyougivethelanguagetosomeonewhostartschangingbytesontheproductiondatabase“toseewhathappens”,youwillgetwhatyouaskedfor.

Willuntrustedlanguagescorruptthedatabase?Thepowertocorruptthedatabaseisdefinitelythere,sincethefunctionsrunasthesystemuserofthedatabaseserverwithfullaccesstothefilesystem.So,ifyoublindlystartwritingintothedatafilesanddeletingimportantlogs,itispossiblethatyourdatabasewillbecorrupted.

Additionaltypesofdenial-of-serviceattacksarealsopossible,suchasusingupallmemoryoropeningallIPports.But,therearewaystooverloadthedatabaseusingplainSQLaswell,sothatpartisnotmuchdifferentfromthetrusteddatabaseaccesswiththeabilitytojustrunarbitraryqueries.

Soyes,youcancorruptthedatabase,butpleasedon’tdoitonaproductionserver.Ifyoudo,youwillbesorry.

Whyuntrusted?PostgreSQL’sabilitytouseanuntrustedlanguageisapowerfulwaytoperformsomenon-traditionalthingsfromdatabasefunctions.CreatingthesefunctionsinaPLisataskofsmallermagnitudethanwritinganextensionfunctioninC.Forexample,afunctiontolookupahostnameforanIPaddressisonlyafewlinesinPL/PythonU:

CREATELANGUAGEplpythonu;

CREATEFUNCTIONgethostbyname(hostnametext)

RETURNSinet

AS$$

importsocket

returnsocket.gethostbyname(hostname)

$$LANGUAGEplpythonuSECURITYDEFINER;

Youcantestitimmediatelyaftercreatingthefunctionbyusingpsql:

hannu=#SELECTgethostbyname('www.postgresql.org');

gethostbyname

----------------

98.129.198.126

(1row)

Creatingthesamefunctioninthemostuntrustedlanguage,C,involveswritingtensoflinesofboilerplatecode,worryingaboutmemoryleaks,andalltheotherproblemscomingfromwritingcodeinalow-levellanguage.WhilewewilllookatextendingPostgreSQLinCinthenextchapter,IrecommendprototypinginaPLlanguageifpossible,andinanuntrustedlanguageifthefunctionneedssomethingthattherestrictedlanguagesdonotoffer.

WhyPL/Python?AllofthesetaskscouldbedoneequallywellusingPL/PerlUorPL/TclU.IchosePL/PythonUmainlybecausePythonisthelanguageIammostcomfortablewith.ThisalsotranslatestohavingwrittensomePL/Pythoncode,whichIplantodiscussandsharewithyouinthischapter.

QuickintroductiontoPL/PythonInthepreviouschapters,wediscussedPL/pgSQLwhichisoneofthestandardprocedurallanguagesdistributedwithPostgreSQL.PL/pgSQLisalanguageuniquetoPostgreSQLandwasdesignedtoaddblocksofcomputationandSQLinsidethedatabase.Whileithasgrowninitsbreadthoffunctionality,itstilllacksthecompletenessofsyntaxofafullprogramminglanguage.PL/PythonallowsyourdatabasefunctionstobewritteninPythonwithallthedepthandmaturityofwritingaPythoncodeoutsidethedatabase.

AminimalPL/PythonfunctionLet’sstartfromtheverybeginning,yetagain:

CREATEFUNCTIONhello(nametext)

RETURNStext

AS$$

return'hello%s!'%name

$$LANGUAGEplpythonu;

Here,weseethatcreatingafunctionstartsbydefiningitasanyotherPostgreSQLfunctionwithaRETURNSdefinitionofatextfield:

CREATEFUNCTIONhello(nametext)

RETURNStext

Thedifferencefromwhatwehaveseenbefore,isthatthelanguagepartisspecifyingplpythonu(thelanguageIDforthePL/PythonUlanguage):


Insidethefunctionbody,itisverymuchanormalPythonfunctionreturningavalueobtainedbythenamepassedasanargumentformattedintoastring'hello%s!',usingthestandardPythonformattingoperator%:

return'hello%s!'%name

Finally,let’stesthowthisworks:

hannu=#SELECThello('world');

hello

---------------

helloworld!

(1row)

Andyes,itreturnsexactlywhatisexpected!

DatatypeconversionsThefirstandlastthingshappeningwhenaPLfunctioniscalledbyPostgreSQL,areconvertingargumentvaluesbetweenthePostgreSQLandPLtypes.ThePostgreSQLtypesneedtobeconvertedtothePLtypesonenteringthefunction,andthenthereturnvalueneedstobeconvertedbackintothePostgreSQLtypes.

ExceptforPL/pgSQL,whichusesPostgreSQL’sownnativetypesincomputations,thePLsarebasedonexistinglanguageswiththeirownunderstandingofwhattypes(integer,string,date,andsoon)are,howtheyshouldbehave,andhowtheyarerepresentedinternally.TheyaremostlysimilartoPostgreSQL’sunderstandingbutquiteoftenarenotexactlythesame.PL/PythonconvertsdatafromPostgreSQLtypestoPythontypes,asshowninthefollowingtable:

PostgreSQL Python2

Python3 Comments

int2,int4 int int

int8 long int

real,double,numeric float float Thismayloseprecisionfornumericvalues.

bytea str bytesNoencodingconversionisdone,norshouldanyencodingbeassumed.

text,char(),varchar(),andothertexttypes

str strOnPython2,thestringwillbeinserverencoding.

OnPython3,itisaunicodestring.

Allothertypes str strPostgreSQL’stypeoutputfunctionisusedtoconverttothisstring.

Insidethefunction,allcomputationisdoneusingPythontypesandthereturnvalueisconvertedbacktoPostgreSQLusingthefollowingrules(theserulesarethedirectquotesfromofficialPL/Pythondocumentationathttp://www.postgresql.org/docs/current/static/plpython-data.html):

WhenthePostgreSQLreturntypeisBoolean,thereturnvaluewillbeevaluatedfortruth,accordingtothePythonrules.Thatis,0andemptystringsarefalse,butnotablyfistrue.WhenthePostgreSQLreturntypeisbytea,thereturnvaluewillbeconvertedtoastring(Python2)orbytes(Python3)usingtherespectivePythonbuilt-ins,withtheresultbeingconvertedbytea.ForallotherPostgreSQLreturntypes,thereturnedPythonvalueisconvertedtoastringusingPython’sbuilt-instr,andtheresultispassedtotheinputfunctionofthePostgreSQLdatatype.

StringsinPython2arerequiredtobeinthePostgreSQLserverencodingwhentheyarepassedtoPostgreSQL.Stringsthatarenotvalidinthecurrentserverencodingwillraise

http://www.postgresql.org/docs/current/static/plpython-data.html

anerror.Butnotallencodingmismatchescanbedetected,sogarbagedatacanstillresultwhenthisisnotdonecorrectly.Unicodestringsareconvertedtothecorrectencodingautomatically,soitcanbesaferandmoreconvenienttousethose.InPython3,allstringsareUnicodestrings.

Inotherwords,anythingbut0,False,andanemptysequence,includingemptystrings'',oradictionarybecomesPostgreSQLfalse.

Onenotableexceptiontothis,isthatthecheckforNoneisdonebeforeanyotherconversions.EvenforBooleans,NoneisalwaysconvertedtoNULLandnottotheBooleanvaluefalse.

Forthebyteatype,thePostgreSQLbytearray,theconversionfromPython’sstringrepresentation,isanexactcopywithnoencodingorotherconversionsapplied.

WritingsimplefunctionsinPL/PythonWritingfunctionsinPL/PythonisnotmuchdifferentinprinciplefromwritingfunctionsinPL/pgSQL.Youstillhavetheexactsamesyntaxaroundthefunctionbodyin$$,andtheargumentname,types,andreturnsallmeanthesamething,regardlessoftheexactPL/languageused.

AsimplefunctionSo,asimpleadd_one()functioninPL/Pythonlookslikethis:

CREATEFUNCTIONadd_one(iint)

RETURNSintAS$$

returni+1;


usm=#SELECTadd_one(2);

add_one

---------

3

(1row)

Itcan’tgetanysimplerthanthat,canit?

WhatyouseehereisthatthePL/PythonargumentsarepassedtothePythoncodeafterconvertingthemtoappropriatetypes,andtheresultispassedbackandconvertedtotheappropriatePostgreSQLtypeforthereturnvalue.

FunctionsreturningarecordToreturnarecordfromaPythonfunction,youcanuse:

AsequenceorlistofvaluesinthesameorderasthefieldsinthereturnrecordAdictionarywithkeysmatchingthefieldsinthereturnrecordAclassortypeinstancewithattributesmatchingthefieldsinthereturnrecord

Herearesamplesofthethreewaystoreturnarecord:

First,usinganinstance:

CREATEORREPLACEFUNCTIONuserinfo(

INOUTusernamename,

OUTuser_idoid,

OUTis_superuserboolean)

AS$$

classPGUser:

def__init__(self,username,user_id,is_superuser):

self.username=username

self.user_id=user_id

self.is_superuser=is_superuser

u=plpy.execute("""\

selectusename,usesysid,usesuper

frompg_user

whereusename='%s'"""%username)[0]

user=PGUser(u['usename'],u['usesysid'],u['usesuper'])

returnuser


Then,alittlesimpleroneusingadictionary:


INOUTusernamename,

OUTuser_idoid,


AS$$

u=plpy.execute("""\


frompg_user


return{'username':u['usename'],'user_id':u['usesysid'],

'is_superuser':u['usesuper']}


Finally,usingatuple:


INOUTusernamename,

OUTuser_idoid,


AS$$

u=plpy.execute("""\


frompg_user


return(u['usename'],u['usesysid'],u['usesuper'])


Notice[0]attheendofu=plpy.execute(...)[0]inalltheexamples.Itistheretoextractthefirstrowoftheresult,asevenforone-rowresultsplpy.executestillreturnsalistofresults.

TipDangerofSQLinjection!

Aswehaveneitherexecutedaprepare()methodandexecutedaexecute()methodwithargumentsafterit,norhaveweusedtheplpy.quote_literal()method(bothtechniquesarediscussedlater)tosafelyquotetheusernamebeforemergingitintothequery,weareopentoasecurityflawknownasSQLinjection.So,makesurethatyouonlylettrusteduserscallthisfunctionorsupplytheusernameargument.

CallingthefunctiondefinedviaanyofthesethreeCREATEcommandswilllookexactlythesame:

hannu=#SELECT*FROMuserinfo('postgres');

username|user_id|is_superuser

----------+---------+--------------

postgres|10|t

(1row)

Itusuallydoesnotmakesensetodeclareaclassinsideafunctionjusttoreturnarecordvalue.Thispossibilityisincludedmostlyforcaseswhereyoualreadyhaveasuitableclass

withasetofattributesmatchingtheonesthefunctionreturns.

TablefunctionsWhenreturningasetfromPL/Pythonfunctions,youhavethreeoptions:

ReturnalistoranyothersequenceofreturntypeReturnaniteratororgeneratorTheyieldkeywordinpythonjustreturnsagenerator

Here,wehavethreewaystogenerateallevennumbersuptotheargumentvalueusingthesedifferentstyles:

First,returningalistofintegers:

CREATEFUNCTIONeven_numbers_from_list(up_toint)

RETURNSSETOFint

AS$$

returnrange(0,up_to,2)


libro=#SELECT*FROMeven_numbers_from_list(10);

even_numbers_from_list

------------------------

0

2

4

6

8

(5rows)

Thelisthere,isreturnedbyabuilt-inPythonfunctioncalledrange,whichreturnsaresultofallevennumbersbelowtheargument.Thisgetsreturnedasatableofintegers,oneintegerperrowfromthePostgreSQLfunction.IftheRETURNSclauseofthefunctiondefinitionwouldsayint[]insteadofSETOFint,thesamefunctionwouldreturnasinglenumberofevenintegersasaPostgreSQLarray.

Thenextfunctionreturnsasimilarresultusingageneratorandreturningboththeevennumberandtheoddonefollowingit.Also,noticethedifferentPostgreSQLsyntaxRETURNSTABLE(...)usedthistimefordefiningthereturnset:

CREATEFUNCTIONeven_numbers_from_generator(up_toint)

RETURNSTABLE(evenint,oddint)

AS$$

return((i,i+1)foriinxrange(0,up_to,2))


libro=#SELECT*FROMeven_numbers_from_generator(10);

even|odd

------+-----

0|1

2|3

4|5

6|7

8|9

(5rows)

Thegeneratorisconstructedusingageneratorexpression(xforxin<seq>).Finally,thefunctionisdefinedusingageneratorusinganexplicityieldsyntax,andyetanotherPostgreSQLsyntaxisusedforreturningSETOFRECORDwiththerecordstructuredefinedthistimebyOUTparameters:

CREATEFUNCTIONeven_numbers_with_yield(up_toint,

OUTevenint,

OUToddint)

RETURNSSETOFRECORD

AS$$

foriinxrange(0,up_to,2):

yieldi,i+1


Theimportantparthere,isthatyoucanuseanyoftheprecedingwaystodefineaPL/Pythonsetreturningfunctionandtheyallworkthesame.Also,youarefreetoreturnamixtureofdifferenttypesforeachrowoftheset:

CREATEFUNCTIONbirthdates(OUTnametext,OUTbirthdatedate)

RETURNSSETOFRECORD

AS$$

return(

{'name':'bob','birthdate':'1980-10-10'},

{'name':'mary','birthdate':'1983-02-17'},

['jill','2010-01-15'],

)


Thisyieldstheresult,asfollows:

hannu=#SELECT*FROMbirthdates();

name|birthdate

------+------------

bob|1980-10-10

mary|1983-02-17

jill|2010-01-15

(3rows)

Asyoucansee,thedatareturningapartofPL/PythonUismuchmoreflexiblethanreturningdatafromafunctionwritteninPL/pgSQL.

RunningqueriesinthedatabaseIfyouhaveeveraccessedadatabaseinPython,youknowthatmostdatabaseadaptersconformtoasomewhatloosestandardcalledPythonDatabaseAPISpecificationv2.0orDBAPI2forshort.Youcanfindthereferenceonlineathttp://legacy.python.org/dev/peps/pep-0249/

ThefirstthingyouneedtoknowaboutdatabaseaccessinPL/Pythonisthatin-databasequeriesdonotfollowthisAPI.

RunningsimplequeriesInsteadofusingthestandardAPI,therearejustthreefunctionsfordoingalldatabaseaccess.Therearetwovariants:plpy.executeforrunningaquery,andplpy.prepare()forturningaquerytextintoaqueryplanorapreparedquery.

Thesimplestwaytodoaqueryiswith:

res=plpy.execute(<querytext>,[<rowcount>])

Thistakesatextualqueryandanoptionalrowcount,andreturnsaresultobject,whichemulatesalistofdictionaries,onedictionaryperrow.

Asanexample,ifyouwanttoaccessafield'name'ofthethirdrowoftheresult,youuse:

res[2]['name']

Theindexis2andnot3becausePythonlistsareindexedstartingfrom0,sothefirstrowisres[0],thesecondrowres[1],andsoon.

UsingpreparedqueriesInanidealworld,thiswouldbeallthatisneeded,butplpy.execute(query,cnt)hastwoshortcomings:

ItdoesnotsupportparametersTheplanforthequeryisnotsaved,requiringthequerytexttobeparsedandrunthroughtheoptimizerateachinvocation

Wewillshowawaytoproperlyconstructaquerystringlater,butformostusessimpleparameterpassingisenough.So,theexecute(query,[maxrows])callbecomesasetoftwostatements:

plan=plpy.prepare(<querytext>,<listofargumenttypes>)

res=plpy.execute(plan,<listofvalues>,[<rowcount>])

Forexample,toqueryifauser‘postgres’isasuperuser,usethefollowing:

plan=plpy.prepare("selectusesuperfrompg_userwhereusename=$1",

["text"])

res=plpy.execute(plan,["postgres"])

printres[0]["usesuper"]

Thefirststatementpreparesthequery,whichparsesthequerystringintoaquerytree,

http://legacy.python.org/dev/peps/pep-0249/

optimizesthistreetoproducethebestqueryplanavailable,andreturnstheprepared_queryobject.Thesecondrowusesthepreparedplantoqueryforaspecificuser’ssuperuserstatus.

Thepreparedplancanbeusedmultipletimes,sothatyoucouldcontinuetoseeifuserbobissuperuser.

res=plpy.execute(plan,["bob"])

printres[0]["usesuper"]

CachingpreparedqueriesPreparingthequerycanbequiteanexpensivestep,especiallyformorecomplexquerieswheretheoptimizerhastochoosefromaratherlargesetofpossibleplans.So,itmakessensetore-useresultsofthisstep,ifpossible.

ThecurrentimplementationofPL/Pythondoesnotautomaticallycachequeryplans(preparedqueries),butyoucandoityourselfeasily.

try:

plan=SD['is_super_qplan']

except:

SD['is_super_qplan']=plpy.prepare("....

plan=SD['is_super_qplan']

<therestofthefunction>

TheglobaldictionarySDisavailabletostoredatabetweenfunctioncalls.Thisvariableisprivatestaticdata.TheglobaldictionaryGDispublicdata,availabletoallPythonfunctionswithinasession.Usewithcare.ThevaluesinSD[]andGD[]onlyliveinsideasingledatabasesession,soitonlymakessensetodothecachingincaseyouhavelong-livedconnections.

WritingtriggerfunctionsinPL/PythonAswithotherPLs,PL/PythonUcanbeusedtowritetriggerfunctions.ThedeclarationofatriggerfunctionisdifferentfromanordinaryfunctionbythereturntypeRETURNSTRIGGER.So,asimpletriggerfunctionthatjustnotifiesthecallerthatitisindeedcalled,lookslikethis:

CREATEORREPLACEFUNCTIONnotify_on_call()

RETURNSTRIGGER

AS$$

plpy.notice('Iwascalled!')


Aftercreatingthisfunction,thetriggercanbetestedonatableusingatriggerfunction:

hannu=#CREATETABLEttable(idint);

CREATETABLE

hannu=#CREATETRIGGERttable_notifyBEFOREINSERTONttableEXECUTE

PROCEDUREnotify_on_call();

CREATETRIGGER

hannu=#INSERTINTOttableVALUES(1);

NOTICE:Iwascalled!

CONTEXT:PL/Pythonfunction"notify_on_call"

INSERT01

Ofcourse,theprecedingtriggerfunctionisquiteuseless,aswillbeanytriggerwithoutknowingwhenandonwhatdatachange,thetriggerwascalled.Allthedataneededbyatrigger,whenitiscalled,ispassedinviathetriggerdictionarycalledTD.InTD,youhavethefollowingvalues:

Key Value

TD["event"]

Theeventthetriggerfunctioniscalledfor;oneofthefollowingstringsiscontainedastheevent:

INSERT,UPDATE,DELETE,orTRUNCATE

TD["when"] OneofBEFORE,AFTER,orINSTEADOF

TD["level"] ROWorSTATEMENT

TD["old"]

Thisisthebefore-commandimageoftherow.Forlow-levelUPDATEandDELETEtriggers,thiscontainsadictionaryforthevaluesofthetriggeringrow,beforethechangeshavebeenmadebythecommand.ItisNoneforothercases.

TD["new"]

Thisistheafter-commandimageoftherow.Forlow-levelINSERTandUPDATEtriggers,thiscontainsadictionaryforthevaluesofthetriggeringrow,afterthechangeshavebeenmadebythecommand.ItisNoneforothercases.

IfyouareinaBEFOREorINSTEADOFtrigger,youcanmakechangestothisdictionaryandthensignalPostgreSQLtousethechangedtuplebyreturningthestringMODIFYfromthetriggerfunction.

TD["name"] ThetriggernamefromtheCREATETRIGGERcommand.

TD["table_name"] Thenameofthetableonwhichthetriggeroccurred.

TD["table_schema"] Theschemaofthetableonwhichthetriggeroccurred.

TD["relid"] Theobjectidentifier(OID)ofthetableonwhichthetriggeroccurred.

TD["args"]IftheCREATETRIGGERcommandincludedarguments,theyareavailablefromTD["args"][0]toTD["args"][n-1].

InadditiontodoinganythingyoucandoinordinaryPL/Pythonfunctions,suchasmodifyingdataintables,writingtofilesandsockets,andsendinge-mails,youcanalsoaffectthebehaviorofthetriggeringcommand.

IfTD["when"]is("BEFORE","INSTEADOF")andTD["level"]=="ROW",youcanreturnSKIPtoaborttheevent.ReturningNoneorOKindicatesthattherowisunmodifiedanditisOKtocontinue.ReturningNoneisalsothedefaultbehaviorforPythonifthefunctiondoesasimplereturnorrunstotheendwithoutareturnstatement,inwhichcase,youdon’tneedtodoanything.

IncaseyouhavemodifiedvaluesintheTD["new"]andyouwantPostgreSQLtocontinuewiththenewvalues,youcanreturnMODIFYtoindicatetoPL/Pythonthatyou’vemodifiedthenewrow.ThiscanonlybedoneifTD["event"]isINSERTorUPDATE,otherwisethereturnvalueisignored.

ExploringtheinputsofatriggerThefollowingtriggerfunctionisusefulwhendevelopingtriggers,sothatyoucaneasilyseewhatthetriggerfunctionisreallygettingwhencalled:

CREATEORREPLACEFUNCTIONexplore_trigger()

RETURNSTRIGGER

AS$$

importpprint

nice_data=pprint.pformat(

(

('TD["table_schema"]',TD["table_schema"]),

('TD["event"]',TD["event"]),

('TD["when"]',TD["when"]),

('TD["level"]',TD["level"]),

('TD["old"]',TD["old"]),

('TD["new"]',TD["new"]),

('TD["name"]',TD["name"]),

('TD["table_name"]',TD["table_name"]),

('TD["relid"]',TD["relid"]),

('TD["args"]',TD["args"]),

)

)

plpy.notice('explore_trigger:\n'+nice_data)


ThisfunctionformatsallthedatapassedtothetriggerinTDusingpprint.pformat,andthensendsittotheclientasastandardPythoninfomessageusingplpy.notice.Fortestingthisout,wecreateasimpletableandthenputanAFTER…FOREACHROW…trigger

usingthisfunctiononthattable:

CREATETABLEtest(

idserialPRIMARYKEY,

datatext,

tstimestampDEFAULTclock_timestamp()

);

CREATETRIGGERtest_explore_trigger

AFTERINSERTORUPDATEORDELETEONtest

FOREACHROW

EXECUTEPROCEDUREexplore_trigger('one',2,null);

Now,wecanexplorewhatthetriggerfunctionactuallygets:

hannu=#INSERTINTOtest(id,data)VALUES(1,'firstrowdata');

NOTICE:explore_trigger:

(('TD["table_schema"]','public'),

('TD["event"]','INSERT'),

('TD["when"]','AFTER'),

('TD["level"]','ROW'),

('TD["old"]',None),

('TD["new"]',

{'data':'firstrowdata','id':1,'ts':'2013-05-1312:04:03.676314'}),

('TD["name"]','test_explore_trigger'),

('TD["table_name"]','test'),

('TD["relid"]','35163'),

('TD["args"]',['one','2','null']))

CONTEXT:PL/Pythonfunction"explore_trigger"

INSERT01

MostofthisisexpectedandcorrespondswelltothetableoftheTDdictionaryvaluesgivenintheprevioustable.Whatmaybealittleunexpected,isthefactthattheargumentsgivenintheCREATETRIGGERstatementareallconvertedtostrings,eventheNULL.Whendevelopingyourowntriggers,eitherinPL/Pythonoranyotherlanguage,itmaybeusefultoputthistriggeronthetableaswell,tocheckthattheinputstothetriggerareasexpected.Forexample,itiseasytoseethatifyouomittheFOREACHROWpart,theTD['old']andTD['new']willbothbeempty,asthetriggerdefinitiondefaultstoFOREACHSTATEMENT.

AlogtriggerNow,wecanputthisknowledgetoworkandwriteatriggerthatlogschangestothetabletoeitherafileortoaspeciallog-collectorprocessoverthenetwork.Loggingtoafileisthesimplestwaytopermanentlylogthechangesintransactionswhichwererolledback.Ifthesewereloggedtoalogtable,theROLLBACKcommandwouldalsoremovethelogrecords.Thismaybeacrucialauditrequirementforyourbusiness.

Ofcourse,thisalsohasadownside.Youwillbeloggingthechangesthatmaynotbepermanentduetothetransactionbeingrolledback.Unfortunately,thisisthepriceyouhavetopayfornotlosingthelogrecords.

CREATEORREPLACEFUNCTIONlog_trigger()

RETURNSTRIGGERAS$$

args=tuple(TD["args"])

ifnotSD.has_key(args):

protocol=args[0]

ifprotocol=='udp':

importsocket

sock=socket.socket(socket.AF_INET,

socket.SOCK_DGRAM)

deflogfunc(msg,addr=args[1],

port=int(args[2]),sock=sock):

sock.sendto(msg,(addr,port))

elifprotocol=='file':

f=open(args[1],'a+')

deflogfunc(msg,f=f):

f.write(msg+'\n')

f.flush()

else:

raiseValueError,'badlogdestinCREATETRIGGER'

SD[args]=logfunc

SD['env_plan']=plpy.prepare("""

selectclock_timestamp(),

txid_current(),

current_user,

current_database()""",[])

logfunc=SD[args]

env_info_row=plpy.execute(SD['env_plan'])[0]

importjson

log_msg=json.dumps(

{'txid':env_info_row['txid_current'],

'time':env_info_row['clock_timestamp'],

'user':env_info_row['current_user'],

'db':env_info_row['current_database'],

'table':'%s.%s'%(TD['table_name'],

TD['table_schema']),

'event':TD['event'],

'old':TD['old'],

'new':TD['new'],

}

)

logfunc(log_msg)


First,thistriggerchecksifitalreadyhasaloggerfunctiondefinedandcachedinthefunction’slocaldictionarySD[].Asthesametriggermaybeusedwithmanydifferentlogdestinations,thelogfunctionisstoredunderthekeyconstructedasaPythontuplefromthetriggerfunctionargumentsintheCREATETRIGGERstatement.WecannotusetheTD["args"]listdirectlyasakey,asPythondictionarykeyshavetobeimmutable,whichalistisnot,butatupleis.

Ifthekeyisnotpresent,meaningthisisthefirstcalltothisparticulartrigger,wehavetocreateanappropriatelogfunctionandstoreit.Todothis,weexaminethefirstargumentforthelogdestinationtype.

Fortheudplogtype,wecreateaUDPsocketforwriting.Then,wedefineafunctionpassinginthissocketandalsotheothertwotriggerargumentsasdefaultargumentsforthe

function.ThisisthemostconvenientwaytocreateaclosureandtobundleafunctionwithsomedatavaluesinPython.

Forthefiletype,wejustopenthisfileintheappendmode(a+)andalsocreatealogfunction.Thelogfunctionwritesamessagetothisfileandflushesthewrite,sothedataiswrittentothefileimmediatelyandnotsometimelaterwhenthewritebufferfillsup(thisisnotpreferableforperformancecriticalsystems).ThelogfunctioncreatedineitherofthesecasesisstoredinSD[tuple(TD["args"])].

Atthispoint,wealsoprepareandsaveaqueryplanforgettingotherdatawewanttologandsavethisinSD['env_plan'].Nowthatwearedonewiththeone-timepreparations,wecanproceedwiththeactualloggingpart,whichisreallyverysimple.

Next,weretrievetheloggingfunction(logfunc=SD[args])andgettherowoftheotherloggeddata:

env_info_row=plpy.execute(SD['env_plan'])[0]

Finally,weconvertalltheloggeddataintooneJSONobject(log_msg=json.dumps({...}))andthenusetheloggingfunctiontosendittothelog,logfunc(log_msg).

Andthat’sit.

Next,let’stestitouttoseehowitworksbyaddinganothertriggertoourtesttablewecreatedearlier:

CREATETRIGGERtest_audit_trigger


FOREACHROW

EXECUTEPROCEDURElog_trigger('file','/tmp/test.json.log');

AnychangestothetabledoneviaINSERT,UPDATE,orDELETEareloggedinto/tmp/test.json.log.Thisfileisinitiallyownedbythesameuserrunningtheserver,usuallypostgres.So,tolookatityouneedtoeitherbethatuserorrootuser,oryouhavetochangethepermissionsonthefilecreatedtoallowreading.

IfyouwanttotesttheUDPloggingpart,youjusthavetodefineanothertriggerwithdifferentarguments:

CREATETRIGGERtest_audit_trigger_udp


FOREACHROW

EXECUTEPROCEDURElog_trigger('udp','localhost',9999);

Ofcourse,youneedsomethingtolistenattheUDPportthere.AminimalistUDPlistenerisprovidedfortestinginthelog_udp_listener.pyfileunderchapter07/logtrigger/.Justrunit,anditprintsanyUDPpacketsreceivedtostdout.

ConstructingqueriesPL/Pythondoesagoodjobofmanagingvaluespassedtopreparedqueryplans,butastandardPostgreSQLqueryplancantakeanargumentinaverylimitednumberofplaces.Sometimes,youmaywanttoconstructwholequeries,notjustpassvaluestopredefinedqueries.Forexample,youcan’thaveanargumentforatablename,orafieldname.

So,howwouldyouproceedifyouwanttoconstructaqueryfromthefunction’sargumentsandbesurethateverythingisquotedproperlyandnoSQLinjectionwouldbepossible?PL/Pythonprovidesthreefunctionstohelpyouwithproperquotingofidentifiersanddata,justforthispurpose.

Thefunctionplpy.quote_ident(nameismeantforquotingidentifiers,thatis,anythingthatnamesadatabaseobjectoritsattributelikeatable,aview,afieldname,orfunctionname.Itsurroundsthenamewithdoublequotesandtakescareofproperlyescapinganythinginsidethestringwhichwouldbreakthequoting:

hannu=#DOLANGUAGEplpythonu$$plpy.notice(plpy.quote_ident(r'5"\"'))

$$;

NOTICE:"5""\"""

CONTEXT:PL/Pythonanonymouscodeblock

DO

Andyes,5"\"isalegaltableorfieldnameinPostgreSQL;youjusthavetoalwaysquoteitifyouuseitinanystatement.

NoteTheDOsyntaxcreatesananonymousblockinsideyourdatabasesession.Itisaveryhandywaytorunsomeprocedurallanguagecodewithoutneedingtocreateafunction.

Theothertwofunctionsareforquotingliteralvalues.Thefunction,plpy.quote_literal(litvalue),isforquotingstringsandplpy.quote_nullable(value_or_none)isforquotingavalue,whichmaybeNone.Bothofthesefunctionsquotestringsinasimilarway,byenclosingtheminsinglequotes(strbecomes'str')anddoublinganysinglequotesorbackslashes:

hannu=#DOLANGUAGEplpythonu$$plpy.notice(plpy.quote_literal(r"\'"))

$$;

NOTICE:E'\\'''

CONTEXT:PL/Pythonanonymouscodeblock

DO

Theonlydifferencebetweenthesetwo,isthatplpy.quote_nullable()canalsotakeavalueNone,whichwillberenderedasastringNULLwithoutanysurroundingquotes.Theargumenttobothofthesehastobeastringoraunicodestring.IfyouwantittoworkwithavalueofanyPythontype,wrappingthevalueinstr(value)usuallyworkswell.

HandlingexceptionsWithanybitofcode,youneedtomakesureyouhandlewhenerrorsoccurandyourPL/Pythonfunctionsarenotanexception.

BeforeVersion9.1ofPostgreSQL,anyerrorinanSQLquerycausedthesurroundingtransactiontoberolledback:

DOLANGUAGEplpythonu$$

plpy.execute('insertintottablevalues(1)')

plpy.execute('fail!')

$$;

ERROR:spiexceptions.SyntaxError:syntaxerroratornear"fail"

LINE1:fail!

^

QUERY:fail!

CONTEXT:Traceback(mostrecentcalllast):

PL/Pythonanonymouscodeblock,line3,in<module>


PL/Pythonanonymouscodeblock

YoucanmanuallyusetheSAVEPOINTattributestocontroltheboundariesoftherolled-backblock,atleastasfarbackasVersion8.4ofPostgreSQL.Thiswillreducetheamountofthetransactionthatisrolledback:

CREATEORREPLACEFUNCTIONsyntax_error_rollback_test()

RETURNSvoid

AS$$


try:

plpy.execute('SAVEPOINTfoo;')



except:

pass



hannu=#SELECTsyntax_error_rollback_test();

syntax_error_rollback_test

---------------------------

(1row)

WhentheSAVEPOINTfoo;commandisexecutedinPL/Python,anSQLerrorwillnotcausefullROLLBACK;butanequivalentofROLLBACKTOSAVEPOINTfoo;,so,onlytheeffectsofcommandsbetweenSAVEPOINTandtheerrorarerolledback:

hannu=#SELECT*FROMttable;

id

----

1

3

(2rows)

StartinginVersion9.1,therearetwoimportantchangesinhowPostgreSQLexceptionsarehandled.IfnoSAVEPOINTorsubtransactionisused,eachinvocationofplpy.prepare()andplpy.execute()isruninitsownsubtransaction,sothatanerrorwillonlyrollbackthissubtransactionandnotallofthecurrenttransaction.Sinceusingaseparatesubtransactionforeachdatabaseinteractioninvolvesextracosts,andyoumaywanttocontrolthesubtransactionboundariesanyway,anewPythoncontextmanager,plpy.subtransaction(),isprovided.

ForanexplanationofPython’scontextmanagers,refertohttp://docs.python.org/library/stdtypes.html#context-manager-types,sothatyoucanusethewithstatementinPython2.6,ornewer,towrapagroupofdatabaseinteractionsinonesubtransactioninamorePythonicway:

hannu=#CREATETABLEtest_ex(iint);

CREATETABLE


plpy.execute('insertintotest_exvalues(1)')

try:

withplpy.subtransaction():



exceptplpy.spiexceptions.SyntaxError:

pass#silentlyignore,avoiddoingthisinprod.code


$$;

DO

hannu=#SELECT*FROMtest_ex;

i

---

1

3

(2rows)

http://docs.python.org/library/stdtypes.html#context-manager-types

AtomicityinPythonWhilethesubtransactionsmanagedatachangesinthePostgreSQLdatabase,thevariablesonthePythonsideofthefencelivetheirseparatelives.Pythondoesnotprovideevenasingle-statementlevelatomicity,asdemonstratedbythefollowing:

>>>a=1

>>>a[1]=a=2

Traceback(mostrecentcalllast):

File"<stdin>",line1,in<module>

TypeError:'int'objectdoesnotsupportitemassignment

>>>a

1

>>>a=a[1]=2

Traceback(mostrecentcalllast):

File"<stdin>",line1,in<module>

TypeError:'int'objectdoesnotsupportitemassignment

>>>a

2

Asyoucansee,itispossiblethatevenasinglemulti-assignmentstatementcanbeexecutedonlyhalfwaythrough.ThismeansthatyouhavetobepreparedtofullymanageyourPythondatayourself.Thefunction,plpy.subtransaction(),won’thelpyouinanywaywithmanagingPythonvariables.

DebuggingPL/PythonFirst,let’sstartbystatingthatthereisnodebuggersupportwhenrunningfunctionsinPL/Python;so,itisagoodideatodevelopanddebugaPL/PythonfunctionasapurePythonfunctionasmuchaspossibleandonlydothefinalintegrationinPL/Python.Tohelpwiththis,youcanhaveasimilarenvironmentinyourPythondevelopmentenvironmentusingtheplpymodule.

JustputthemoduleinyourpathanddoimportplpybeforeyoutryrunningyourprospectivePL/PythonUfunctionsinanordinaryinterpreter.Ifyouuseanyoftheplpy.execute(...)orplpy.prepare()functions,youalsoneedtosetupadatabaseconnectionbeforeusingthesebycallingplpy.connect(<connectstring>).

Usingplpy.notice()totrackthefunction’sprogressThedebuggingtechnologyIusemostofteninanylanguage,isprintingoutintermediatevaluesasthefunctionprogresses.Iftheprintoutrollspasttoofast,youcanslowitdownbysleepingasecondortwoaftereachprint.

InstandardPython,itwouldlooklikethis:

deffact(x):

f=1

while(x>0):

f=f*x

x=x–1

print'f:%d,x:%d'%(f,x)

returnf

Itwillprintoutallintermediatevaluesforfandxasitruns:

>>>fact(3)

f:3,x:2

f:6,x:1

f:6,x:0

6

IfyoutrytouseprintinaPL/Pythonfunction,youwilldiscoverthatnothingisprinted.Infact,thereisnosinglelogicalplacetoprinttowhenrunningapluggablelanguageinsideaPostgreSQLserver.

TheclosestthingtoprintinPL/Pythonisthefunctionplpy.notice(),whichsendsaPostgreSQLNOTICEtotheclientandalsototheserverlogiflog_min_messagesissettothevaluenoticeorsmaller.

CREATEFUNCTIONfact(xint)RETURNSint

AS$$

globalx

f=1

while(x>0):

f=f*x

x=x-1

plpy.notice('f:%d,x:%d'%(f,x))

returnf


Runningthisismuchmoreverbosethantheversionwithprint,becauseeachNOTICEalsoincludesinformationabouttheCONTEXTfromwheretheNOTICEcomes:

hannu=#SELECTfact(3);

NOTICE:f:3,x:2

CONTEXT:PL/Pythonfunction"fact"

NOTICE:f:6,x:1


NOTICE:f:6,x:0


fact

------

6

(1row)

TipPL/PythonUfunctionargumentsarepassedinasglobals

Ifyoucomparedthefact(x)functioninPythonandPL/Python,younoticedanextralineatthebeginningofthePL/Pythonfunction:

globalx

ThisisneededtoovercomeanimplementationdetailthatoftensurprisesPL/PythonUdevelopers;thefunctionargumentsarenotthefunctionargumentsinthePythonsenseandneitheraretheylocals.Theyarepassedinasvariablesinthefunction’sglobalscope.

UsingassertSimilartoordinaryPythonprogramming,youcanalsousePython’sassertstatementtocatchconditionswhichshouldnothappen:

CREATEORREPLACEFUNCTIONfact(xint)

RETURNSint

AS$$

globalx

assertx>=0,"argumentmustbeapositiveinteger"

f=1

while(x>0):

f=f*x

x=x-1

returnf


Totestthis,callfact()withanegativenumber:

hannu=#SELECTfact(-1);

ERROR:AssertionError:argumentmustbeapositiveinteger

CONTEXT:Traceback(mostrecentcalllast):

PL/Pythonfunction"fact",line3,in<module>

assertx>=0,"argumentmustbeapositiveinteger"

PL/Pythonfunction"fact"

YouwillgetamessageaboutAssertionError,togetherwiththelocationofthefailinglinenumber.

Redirectingsys.stdoutandsys.stderrIfallthecodeyouneedtodebugisyourown,theprecedingtwotechniqueswillcovermostofyourneeds.However,whatdoyoudoincaseswhereyouusesomethirdpartylibrarieswhichprintoutdebuginformationtosys.stdoutand/orsys.stderr?

Well,inthosecasesyoucanreplacePython’ssys.stdoutandsys.stdinwithyourownpseudofileobjectthatstoreseverythingwrittenthereforlaterretrieval.Hereisapairoffunctions,thefirstofwhichdoesthecapturingofsys.stdoutoruncapturingifitiscalledwiththeargument,do_capturesettofalse,andthesecondonereturnseverythingcaptured:

CREATEORREPLACEFUNCTIONcapture_stdout(do_capturebool)

RETURNStext

AS$$

importsys

ifdo_capture:

try:

sys.stdout=GD['stdout_to_notice']

exceptKeyError:

classWriteAsNotice:

def__init__(self,old_stdout):

self.old_stdout=old_stdout

self.printed=[]

defwrite(self,s):

self.printed.append(s)

defread(self):

text=''.join(self.printed)

self.printed=[]

returntext

GD['stdout_to_notice']=WriteAsNotice(sys.stdout)

sys.stdout=GD['stdout_to_notice']

return"sys.stdoutcaptured"

else:

sys.stdout=GD['stdout_to_notice'].old_stdout

return"restoredoriginalsys.stdout"


CREATEORREPLACEFUNCTIONread_stdout()

RETURNStext

AS$$

returnGD['stdout_to_notice'].read()


Hereisasamplesessionusingtheprecedingfunctions:

hannu=#SELECTcapture_stdout(true);

capture_stdout

---------------------

sys.stdoutcaptured

(1row)


print'TESTINGsys.stdoutCAPTURING'

importpprint

pprint.pprint({'a':[1,2,3],'b':[4,5,6]})

$$;

DO

hannu=#SELECTread_stdout();

read_stdout

----------------------------------

TESTINGsys.stdoutCAPTURING+

{'a':[1,2,3],'b':[4,5,6]}+

(1row)

Thinkingoutofthe“SQLdatabaseserver”boxWe’llwrapupthechapteronPL/PythonwithacoupleofsamplePL/PythonUfunctionsfordoingsomethingsyouwouldnotusuallyconsiderdoinginsidethedatabasefunctionortrigger.

GeneratingthumbnailswhensavingimagesOurfirstexample,usesPython’spowerfulPythonImagingLibrary(PIL)moduletogeneratethumbnailsofuploadedphotos.Foreaseofinterfacingwithvariousclientlibraries,thisprogramtakestheincomingimagedataasaBase64encodedstring:

CREATEFUNCTIONsave_image_with_thumbnail(image64text)

RETURNSint

AS$$

importImage,cStringIO

size=(64,64)#thumbnailsize

#convertbase64encodedtexttobinaryimagedata

raw_image_data=image64.decode('base64')

#createapseudo-filetoreadimagefrom

infile=cStringIO.StringIO(raw_image_data)

pil_img=Image.open(infile)

pil_img.thumbnail(size,Image.ANTIALIAS)

#createastreamtowritethethumbnailto

outfile=cStringIO.StringIO()

pil_img.save(outfile,'JPEG')

raw_thumbnail=outfile.getvalue()

#storeresultintodatabaseandreturnrowid

q=plpy.prepare('''

INSERTINTOphotos(image,thumbnail)

VALUES($1,$2)

RETURNINGid''',('bytea','bytea'))

res=plpy.execute(q,(raw_image_data,raw_thumbnail))

#returncolumnidoffirstrow

returnres[0]['id']


ThePythoncodeismoreorlessastraightrewritefromthePILtutorial,exceptthatthefilestoreadtheimagefrom,andwritethethumbnailimageto,arereplacedwithPython’sstandardfile-likeStringIOobjects.Forallthistowork,youneedtohavePILinstalledonyourdatabaseserverhost.

InDebian/Ubuntu,thiscanbedonebyrunningsudoapt-getinstallpython-imaging.OnmostmodernLinuxdistributions,analternativeistousePython’sownpackagedistributionsystembyrunningsudoeasy_installPIL.

Sendingane-mailThenextsampleisafunctionforsendinge-mailsfrominsideadatabasefunction:

CREATEORREPLACEFUNCTIONsend_email(

sendertext,—sendere-mail

recipientstext,—comma-separatedlistofrecipientaddresses

subjecttext,—emailsubject

messagetext,—textofthemessage

smtp_servertext—SMTPservertouseforsending

)RETURNSvoid

AS$$

importsmtplib;

msg="From:%s\r\nTo:%s\r\nSubject:%s\r\n\r\n%s"%\

(sender,recipients,subject,message)

recipients_list=[r.strip()forr

inrecipients.split(',')]

server=smtplib.SMTP(smtp_server)

server.sendmail(sender,recipients_list,msg)

server.quit()


test=#SELECTsend_email('[email protected]','[email protected]','test

subject','message','localhost');

Thisfunctionformatsamessage(msg=""),convertsacomma-separatedTo:addressintoalistofe-mailaddresses(recipients_list=[r.strip()...),connectstoaSMTPserver,andthenpassesthemessagetotheSMTPserverfordelivery.

Tousethisfunctioninaproductionsystem,itwouldprobablyrequireabitmorecheckingontheformatsandsomeextraerrorhandling,incasesomethinggoeswrong.YoucanreadmoreaboutPython’ssmtplibathttp://docs.python.org/library/smtplib.html.

http://docs.python.org/library/smtplib.html

ListingdirectorycontentsHereisanotherinterestingusecaseforanuntrustedlanguage.Thefunctionbelowcanlistthecontentsofadirectoryinyoursystem:

CREATEORREPLACEFUNCTIONlist_folder(

directoryVARCHAR—directorythatwillbewalked

)RETURNSSETOFVARCHAR

AS$$

importos;

file_paths=[];

#Walkthetree.

forroot,directories,filesinos.walk(directory):

forfilenameinfiles:

#Jointhetwostringsinordertoformthefullfilepath.

filepath=os.path.join(root,filename)

file_paths.append(filepath)#Addittothelist.

returnfile_paths


Letusnowtryandrunthefunction:

test_db=#SELECTlist_folder('/usr/local/pgsql/bin');

list_folder

-------------------------------------

/usr/local/pgsql/bin/clusterdb

/usr/local/pgsql/bin/createdb

/usr/local/pgsql/bin/createlang

/usr/local/pgsql/bin/createuser

/usr/local/pgsql/bin/dropdb

/usr/local/pgsql/bin/droplang

/usr/local/pgsql/bin/dropuser

/usr/local/pgsql/bin/ecpg

/usr/local/pgsql/bin/initdb

/usr/local/pgsql/bin/pg_basebackup

/usr/local/pgsql/bin/pg_config

/usr/local/pgsql/bin/pg_controldata

/usr/local/pgsql/bin/pg_ctl

/usr/local/pgsql/bin/pg_dump

/usr/local/pgsql/bin/pg_dumpall

/usr/local/pgsql/bin/pg_isready

/usr/local/pgsql/bin/pg_receivexlog

/usr/local/pgsql/bin/pg_resetxlog

/usr/local/pgsql/bin/pg_restore

/usr/local/pgsql/bin/postgres

/usr/local/pgsql/bin/postmaster

/usr/local/pgsql/bin/psql

/usr/local/pgsql/bin/reindexdb

/usr/local/pgsql/bin/vacuumdb

(24rows)

ThefunctionaboveusesthePythonosmoduleandwalksthedirectorytree,top-down.Thisfunctionwillnotwalkdownintosymboliclinksthatresolvetodirectories.Theerrors

areignoredbydefault.YoucanlearnmoreabouthowPython’sos.walk()behavesinPython2’s(sincethatiswhattheexampleuses)documentationherehttps://docs.python.org/2/library/os.html.

https://docs.python.org/2/library/os.html

SummaryInthischapter,wesawthatitisrelativelyeasytodothingswaybeyondwhatasimpleSQLdatabaseservernormallysupports;thankstoitspluggablelanguage’ssupport.

Infact,youcandoalmostanythinginthePostgreSQLserverthatyoucoulddoinanyotherapplicationserver.Hopefully,thischapterjustscratchedthesurfaceofwhatcanbedoneinsideaPostgreSQLserver.

Inthenextchapter,wewilllearnaboutwritingPostgreSQL’smoreadvancedfunctionsinC.ThiswillgiveyoudeeperaccesstoPostgreSQL,allowingyoutouseaPostgreSQLserverformuchmorepowerfulthings.

Chapter9.WritingAdvancedFunctionsinCInthepreviouschapter,weintroducedyoutothepossibilitiesofuntrustedpluggablelanguagesbeingavailabletoaPostgreSQLdevelopertoachievethingsimpossibleinmostotherrelationaldatabases.

Whileusingapluggablescriptinglanguageisenoughforalargeclassofproblems,therearetwomaincategories,wheretheymayfallshort:performanceanddepthoffunctionality.

MostscriptinglanguagesarequiteabitslowerthanoptimizedCcodewhenexecutingthesamealgorithms.Forasinglefunction,thismaynotbethecasebecausecommonthingssuchasdictionarylookupsorstringmatchinghavebeenoptimizedsowellovertheyears.Butingeneral,Ccodewillbefasterthanscriptedcode.Also,incaseswherethefunctioniscalledmillionsoftimesperquery,theoverheadofactuallycallingthefunctionandconvertingtheargumentsandreturnvaluestoandfromthescriptinglanguagecounterpartscanbeasignificantportionoftheruntime.

ThesecondpotentialproblemwithpluggablelanguagesisthatmostofthemjustdonotsupportthefullrangeofpossibilitiesthatareprovidedbyPostgreSQL.ThereareafewthingsthatsimplycannotbecodedinanythingelsebutC.Forexample,whenyoudefineacompletelynewtypeforPostgreSQL,thetypeinputandoutputfunctions,whichconvertthetype’stextrepresentationtointernalrepresentationandback,needtohandlePostgreSQL’spseudotypecstring.ThisisbasicallytheCstringorazero-terminatedstring.ReturningcstringissimplynotsupportedbyanyofthePLlanguagesincludedinthecoredistribution,atleastnotasofPostgreSQLVersion9.3.ThePLlanguagesalsodonotsupportpseudotypesANYELEMENT,ANYARRAYandespecially“any”VARIADIC.

Inthefollowingsections,wewillgostep-by-stepthroughwritingsomePostgreSQLextensionfunctionsinincreasingcomplexityinC.

Wewillstartfromthesimplestadd2argumentsfunctionwhichisquitesimilartotheoneinPostgreSQLmanual,butwewillpresentthematerialinadifferentorder.So,settingupthebuildenvironmentcomesearlyenoughsothatyoucanfollowushands-onfromtheverybeginning.

Afterthat,wewilldescribesomeimportantthingstobeawareofwhendesigningandwritingcodethatrunsinsidetheserver,suchasmemorymanagement,executingqueries,andretrievingresults.

AsthetopicofwritingC-languagePostgreSQLfunctionscanbequitelargeandourspaceforthistopicislimited,wewilloccasionallyskipsomeofthedetailsandreferyoutothePostgreSQLmanualforextrainformationandspecifications.WearealsolimitingthissectiontoreferencePostgreSQL9.3.Whilemostthingswillworkperfectlyfineacrossversions,therearereferencestopathsthatwillbespecifictoaversion.

ThesimplestCfunction–return(a+b)Let’sstartwithasimplefunction,whichtakestwointegerargumentsandreturnsthesumofthese.Wefirstpresentthesourcecodeandthenwillmoveontoshowyouhowtocompileit,loaditintoPostgreSQL,andthenuseitasanynativefunction.

add_func.cACsourcefileimplementingadd(int,int)returnsintfunctionlookslikethefollowingcodesnippet:

#include"postgres.h"

#include"fmgr.h"

PG_MODULE_MAGIC;

PG_FUNCTION_INFO_V1(add_ab);

Datum

add_ab(PG_FUNCTION_ARGS)

{

int32arg_a=PG_GETARG_INT32(0);

int32arg_b=PG_GETARG_INT32(1);

PG_RETURN_INT32(arg_a+arg_b);

}

Let’sgooverthecodeexplainingtheuseofeachsegment:

#include"postgres.h":ThisincludesmostofthebasicdefinitionsanddeclarationsneededforwritinganyCcodeforrunninginPostgreSQL.#include"fmgr.h":ThisincludesthedefinitionsforPG_*macrosusedinthiscode.PG_MODULE_MAGIC;:Thisisa“magicblock”definedinfmgr.h.ThisblockisusedbytheservertoensurethatitdoesnotloadcodecompiledbyadifferentversionofPostgreSQL,potentiallycrashingtheserver.ItwasintroducedinVersion8.2ofPostgreSQL.IfyoureallyneedtowritecodewhichcanalsobecompiledforPostgreSQLversionsbefore8.2youneedtoputthisbetween#ifdefPG_MODULE_MAGIC/#endif.YouseethisalotinsamplesavailableontheInternet,butyouprobablywillnotneedtodotheifdefforanynewcode.Thelatestpre-8.2Versionbecameofficiallyobsolete(thatisunsupported)inNovember2010,andeven8.2communitysupportendedinDecember2011.PG_FUNCTION_INFO_V1(add_ab);:ThisintroducesthefunctiontoPostgreSQLasVersion1callingaconventionfunction.Withoutthisline,itwillbetreatedasanold-styleVersion0function.(SeetheinformationboxfollowingtheVersion0reference.)Datum:ThisisthereturntypeofaC-languagePostgreSQLfunction.add_ab(PG_FUNCTION_ARGS):Thefunctionnameisadd_abandtherestareitsarguments.ThePG_FUNCTION_ARGSdefinitioncanrepresentanynumberofargumentsandhastobepresent,evenforafunctiontakingnoarguments.int32arg_a=PG_GETARG_INT32(0);:YouneedtousethePG_GETARG_INT32(<argnr>)macro(orcorrespondingPG_GETARG_xxx(<argnr>)forotherargumenttypes)togettheargumentvalue.Theargumentsarenumberedstartingfrom0.int32arg_b=PG_GETARG_INT32(1);:Similartothepreviousdescription.PG_RETURN_INT32(arg_a+arg_b);:Finally,youusethePG_RETURN_<rettype>(<retvalue>)macrotobuildandreturnasuitablereturnvalue.

Youcouldalsohavewrittenthewholefunctionbodyasthefollowingcode:

PG_RETURN_INT32(PG_GETARG_INT32(0)+PG_GETARG_INT32(1));

But,itismuchmorereadableaswritten,andmostlikelyagoodoptimizingCcompilerwillcompilebothintoanequivalentlyfastcode.

Mostcompilerswillissueawarningmessageas:warning:nopreviousprototypefor'add_ab'fortheprecedingcode,soitisagoodideatoalsoputaprototypeforthefunctioninthefile:

Datumadd_ab(PG_FUNCTION_ARGS);

Theusualplacetoputit,isjustbeforethecodelinePG_FUNCTION_INFO_V1(add_ab);.

NoteWhiletheprototypeisnotstrictlyrequired,itenablesmuchcleanercompileswithnowarnings.

Version0callconventionsThereisanevensimplerwaytowritePostgreSQLfunctionsinC,calledtheVersion0callingconventions.Theprecedinga+bfunctioncanbewrittenasthefollowingcode:

intadd_ab(intarg_a,intarg_b)

{

returnarg_a+arg_b;

}

Version0isshorterforverysimplefunctions,butitisseverelylimitedformostotherusages—youcan’tdoevensomebasicthingssuchascheckingifapassbyvalueargumentisnull,returnasetofvalues,orwriteaggregatefunctions.Also,Version0doesnotautomaticallytakecareofhidingmostdifferencesofpassbyvalueandpassbyreferencetypesthatVersion1does.Therefore,itisbettertojustwriteallyourfunctionsusingVersion1callingconventionsandignorethefactthatVersion0evenexists.

Fromthispointforward,weareonlygoingtodiscussVersion1callingconventionsforaCfunction.

NoteIncaseyouareinterested,thereissomemoreinformationonVersion0athttp://www.postgresql.org/docs/current/static/xfunc-c.html#AEN50495,inthesectiontitled35.9.3.Version0CallingConventions.

http://www.postgresql.org/docs/current/static/xfunc-c.html#AEN50495

MakefileThenextstepiscompilingandlinkingthe.csourcefileintoaformthatcanbeloadedintothePostgreSQLserver.ThiscanallbedoneasaseriesofcommandsdefinedinaMakefilefunction.

ThePostgreSQLmanualhasacompletesectionaboutwhichflagsandincludedpathsyoushouldpassoneachofthesupportedplatforms,andhowtodeterminecorrectpathsforincludingfilesandlibraries.

Fortunately,allofthisisalsoautomatednicelyfordevelopersviathePostgreSQLextensionbuildinginfrastructure—orPGXSforshort—whichmakesthisreallyeasyformostmodules.

NoteDependingonwhichversionofPostgreSQLyouhaveinstalled,youmayneedtoaddthedevelopmentpackageforyourplatform.Theseareusuallythe-devor-develpackages.

Now,let’screateourMakefilefunction.Itwilllooklikethefollowingcode:

MODULES=add_func

PG_CONFIG=pg_config

PGXS:=$(shell$(PG_CONFIG)--pgxs)

include$(PGXS)

Andyoucancompileandlinkthemodulebysimplyrunningmake:

$make

gcc…-c-oadd_func.oadd_func.c

gcc…-oadd_func.soadd_func.o

rmadd_func.o

Here,“…”standsforquitesomeamountofflags,includes,andlibrariesaddedbyPGXS.

ThisproducesadynamicallyloadablemoduleinthecurrentdirectorywhichcanbeuseddirectlybyPostgreSQL,ifyourserverhasaccesstothisdirectory,whichmaybethecaseonadevelopmentserver.

Fora“standard”server,asinstalledbyyourpackagemanagementsystem,youwillneedtoputthemoduleinastandardplace.ThiscanbedoneusingthePGXSaswell.

Yousimplyexecutesudomakeinstallandeverythingwillbecopiedtotherightplace,$sudomakeinstall:

[sudo]passwordforhannu:

/bin/mkdir-p'/usr/lib/postgresql/9.3/lib'

/bin/sh

/usr/lib/postgresql/9.3/lib/pgxs/src/makefiles/../../config/install-sh-c-

m755add_func.so'/usr/lib/postgresql/9.3/lib/'

CREATEFUNCTIONadd(int,int)Youarejustonestepawayfrombeingabletousethisfunctioninyourdatabase.YoujustneedtointroducethemoduleyoujustcompiledtoaPostgreSQLdatabaseusingtheCREATEFUNCTIONstatement.

Ifyoufollowedthesamplesuptothispoint,thefollowingstatementisallthatisneeded,alongwithadjustingthepathappropriatelytowherePostgreSQLisinstalledonyourserver:

hannu=#CREATEFUNCTIONadd(int,int)

hannu-#RETURNSint

hannu-#AS'/usr/lib/postgresql/9.3/lib/add_func','add_ab_null'

hannu-#LANGUAGECSTRICT;

CREATEFUNCTION

Andvoilá—youhavecreatedyourfirstPostgreSQLC-languageextensionfunction:

hannu=#selectadd(1,2);

add

-----

3

(1row)

add_func.sql.inWhilewhatwejustcoveredisallthatisneededtohaveaCfunctioninyourdatabase,itisoftenmoreconvenienttoputtheprecedingCREATEFUNCTIONstatementinanSQLfile.

YouusuallydonotknowthefinalpathofwherePostgreSQLisinstalledwhenwritingthecode,especiallyinthelightofrunningonmultipleversionsofPostgreSQLand/oronmultipleoperationsystems.Herealso,PGXScanhelp.

Youneedtowriteafilecalledadd_funcs.sql.inasfollows:

CREATEFUNCTIONadd(int,int)RETURNSint

AS'MODULE_PATHNAME','add_ab'

LANGUAGECSTRICT;

ThenaddthefollowinglineinyourMakefilefunctionrightaftertheMODULES=…line:

DATA_built=add_funcs.sql

Now,whenrunningmake,theadd_funcs.sql.iniscompiledintoafileadd_funcs.sqlwithMODULE_PATHNAMEreplacedbytherealpathwherethemodulewillbeinstalled.

[add_func]$make

sed's,MODULE_PATHNAME,$libdir/add_func,g'add_func.sql.in>add_func.sql

Also,sudomakeinstallwillcopythegenerated.sqlfileintothedirectorywithother.sqlfilesforextensions,asshown:

$sudomakeinstall

/usr/bin/mkdir-p'/usr/lib/postgresql/9.3/share/contrib'

/usr/bin/mkdir-p'/usr/lib/postgresql/9.3/lib'

/bin/sh


m644add_func.sql'/usr/lib/postgresql/9.3/share/contrib/'

/bin/sh


m755add_func.so'/usr/lib/postgresql/9.3/lib/'

Afterthis,theintroductionofyourCfunctionstoaPostgreSQLdatabaseisassimpleashannu=#\i/usr/lib/postgresql/9.3/share/contrib/add_func.sql:

CREATEFUNCTION

Thepath/usr/lib/postgresql/9.3/share/contrib/toadd_funcs.sqlneedstobelookedupfromtheoutputofthemakeinstallcommand.

NoteThereisamuchcleanerwaytopackageupyourcodecalledExtensionswhereyoudon’tneedtolookupforanypathsandtheprecedingstepwouldjustbeasfollows:CREATEEXTENSIONchap8_add;

Butitisrelativelymorecomplextosetup,sowearenotexplainingithere.Chapter13,PublishingYourCodeasPostgreSQLExtensionsdedicatedtoextensionsappearslaterinthisbook.

SummaryforwritingaCfunctionWritingaCfunctionusedinPostgreSQLisastraightforwardprocess:

1. WritetheCcodeinmodulename.c.2. WritetheSQLcodeforCREATEFUNCTIONinmodulename.sql.in.3. WriteaMakefilefunction.4. RunmaketocompileaCfileandgeneratemodulename.sql.5. Runsudomakeinstalltoinstallthegeneratedfiles.6. Runthegeneratedmodulename.sqlinyourtargetdatabase:

hannu#\i/<path>/modulename.sql

NoteYoumustruntheSQLcodeinanydatabaseyouwanttouseyourfunction.Ifyouwantallyournewdatabasestohaveaccesstoyournewlygeneratedfunction,addthefunctiontoyourtemplatedatabasebyrunningthemodulename.sqlfileindatabasetemplate1oranyotherdatabaseyouareexplicitlyspecifyingintheCREATEDATABASEcommand.

Youmayhavenoticedthatwhilecreatingthefunction,youspecifiedthenameoftheloadableobjectfileandthenameoftheCfunction.WhentheSQLfunctioniscalledforthefirsttime,thedynamicloaderloadstheobjectfileinmemory.Ifyouwouldlikesomeobjectfilestobepreloadedatserverstartup,youshouldspecifytheminthePostgreSQLshared_preload_librariesconfigurationparameter.

Addingfunctionalitytoadd(int,int)Whileourfunctionworks,itaddsnothingintheprecedingcodejustusingSELECTA+B.ButfunctionswritteninCarecapableofsomuchmore.Solet’sstartaddingsomemorefunctionalitytoourfunction.

SmarthandlingofNULLargumentsNoticetheuseofaSTRICTkeywordintheCREATEFUNCTIONadd(inta,intb)inthepreviouslymentionedcode.ThismeansthatthefunctionwillnotbecalledifanyoftheargumentsareNULL,butinsteadNULLisreturnedstraightaway.ThisissimilartohowmostPostgreSQLoperatorswork,includingthe+signwhenaddingtwointegers—ifanyoftheargumentsareNULLthecompleteresultisNULLaswell.

Next,wewillextendourfunctiontobesmarteraboutNULLinputsandactlikePostgreSQL’ssum()aggregatefunction,whichignoresNULLvaluesininputsandstillproducesthesumofallnon-nullvalues.

Forthis,weneedtodotwothings:

MakesurethatthefunctioniscalledwheneitheroftheargumentsareNULL.HandleNULLargumentsbyeffectivelyconvertingaNULLargumentto0andreturningNULLonlyincaseswherebothargumentsarenull.

Thefirstoneiseasy—justleaveouttheSTRICTkeywordwhendeclaringthefunction.ThelatteronealsoseemseasyaswejustleaveoutSTRICTandletthefunctionexecute.Forafunctionwithintarguments,thisalmostseemstodothetrick.AllNULLvaluesshowupas0s,andtheonlythingyoumisswillbereturningNULLifbothargumentsareNULL.

Unfortunately,thisonlyworksbycoincidenceandisnotguaranteedtoworkinfutureversions.Worsestill,ifyoudoitthesamewayforpassbyreferencetypes,itwillcausePostgreSQLtocrashonnullpointerreferences.

Next,weshowyouhowtodoitproperly.Weneednowtodotwothings:recordifwehaveanynon-nullvaluesandaddallthenon-nullvalueswesee:

PG_FUNCTION_INFO_V1(add_ab_null);

Datum

add_ab_null(PG_FUNCTION_ARGS)

{

int32not_null=0;

int32sum=0;

if(!PG_ARGISNULL(0)){

sum+=PG_GETARG_INT32(0);

not_null=1;

}

if(!PG_ARGISNULL(1)){

sum+=PG_GETARG_INT32(1);

not_null=1;

}

if(not_null){

PG_RETURN_INT32(sum);

}

PG_RETURN_NULL();

}

Thisindeeddoeswhatweneed:

hannu=#CREATEFUNCTIONadd(int,int)RETURNSint

AS'$libdir/add_func','add_ab_null'

LANGUAGEC;

CREATEFUNCTION

hannu=#SELECTadd(NULL,NULL)asmust_be_null,add(NULL,1)as

must_be_one;

-[RECORD1]+--

must_be_null|

must_be_one|1

AchievingthesameresultusingstandardPostgreSQLstatements,functions,andoperatorswouldbemuchmoreverbose:

hannu=#SELECT(casewhen(aisnull)and(bisnull)

hannu(#thennull

hannu(#elsecoalesce(a,0)+coalesce(b,0)

hannu(#end)

hannu-#FROM(select1::intasa,null::intasb)s;

-[RECORD1]

case|1

Inadditiontorestructuringthecode,wealsointroducedtwonewmacrosPG_ARGISNULL(<argnr>)forcheckingiftheargument<argnr>isNULLandPG_RETURN_NULL()forreturningNULLfromafunction.

NotePG_RETURN_NULL()isdifferentfromPG_RETURN_VOID().Thelatterisforusingfunctionswhicharedeclaredtoreturnpseudotypevoidorinotherwords,nottoreturnanything.

WorkingwithanynumberofargumentsAftertherewritetohandleNULLvalues,itseemsthatwithjustalittlemoreeffort,wecouldmakeitworkwithanynumberofarguments.Justmovethefollowingcodeinsidethefor(;;)cycleovertheargumentsandwearedone:

if(!PG_ARGISNULL(<N>)){

sum+=PG_GETARG_INT32(<N>);

not_null=1;

}

Actually,makingthecodeuseanarrayinsteadofasimpletypeisnotthatsimpleafterall.Tomakethingsmoredifficult,thereisnoinformationorsamplecodeonhowtoworkwitharraysintheofficialPostgreSQLmanualforC-languageextensionfunctions.Thelinebetween“supported”and“unsupported”whenwritingC-languagefunctionsisquiteblurred,andtheprogrammerdoingsoisexpectedtobeabletofiguresomethingsoutindependently.

ThebrightsideisthatthefriendlyfolksatthePostgreSQLmailinglistsareusuallyhappytohelpyououtiftheyseethatyourquestionisaseriousoneandthatyouhavemadesomeefforttofigureoutthebasicstuffyourself.

Toseehowargumentsofarraytypesarehandled,youhavetostartdiggingaroundontheInternetand/orinthebackendcode.Oneplacewhereyoucanfindasampleisthecontrib/hstore/moduleinthePostgreSQLsourcecode.ThecontribmodulesareagreatreferenceforexamplesofofficiallysupportedextensionmodulesfromPostgreSQL.

Thoughthecodetheredoesnotdoexactlywhatweneed—itworksontext[]andnotint[]—itiscloseenoughtofigureoutwhatisneeded,bysupplyingthebasicstructureofarrayhandlingandsampleusageofsomeutilitymacrosandfunctions.

Aftersomediggingaroundinthebackendcodeanddoingsomewebsearches,itisnotveryhardtocomeupwithacodeforintegerarrays.

So,hereistheCcodeforafunctionwhichsumsallnon-nullelementsinitsargumentarray.Thecodeshouldbeaddedtoadd_func.c:

#include"utils/array.h"//arrayutilityfunctionsandmacros

#include"catalog/pg_type.h"//forINT4OID

PG_MODULE_MAGIC;

Datumadd_int32_array(PG_FUNCTION_ARGS);

PG_FUNCTION_INFO_V1(add_int32_array);

Datum

add_int32_array(PG_FUNCTION_ARGS)

{

ArrayType*input_array;

int32sum=0;

boolnot_null=false;

//variablesfor"deconstructed"array

Datum*datums;

bool*nulls;

intcount;

//forloopcounter

inti;

input_array=PG_GETARG_ARRAYTYPE_P(0);

//checkthatwedoindeedhaveaone-dimensionalintarray

Assert(ARR_ELEMTYPE(input_array)==INT4OID);

if(ARR_NDIM(input_array)>1)

ereport(ERROR,

(errcode(ERRCODE_ARRAY_SUBSCRIPT_ERROR),

errmsg("1-dimensionalarrayneeded")));

deconstruct_array(input_array,//one-dimensionalarray

INT4OID,//ofintegers

4,//sizeofintegerinbytes

true,//int4ispass-byvalue

'i',//alignmenttypeis'i'

&datums,&nulls,&count);//resulthere

for(i=0;i<count;i++){

//firstcheckandignorenullelements

if(nulls[i])

continue;

//accumulateandremembertherewerenon-nullvalues

sum+=DatumGetInt32(datums[i]);

not_null=true;

}

if(not_null)

PG_RETURN_INT32(sum);

PG_RETURN_NULL();

}

So,whatnewthingsareneededforhandlingarraytypesasarguments?First,youneedtoincludedefinitionsforarrayutilityfunctions:

#include"utils/array.h"

Next,youneedapointertoyourarray:

ArrayType*input_array;

Noticethatthereisnospecificarray-of-integerstypebutjustagenericArrayType,whichisusedforanyarray.

Toinitializethearrayfromthefirstargument,youuseanalreadyfamiliarlookingmacro:

input_array=PG_GETARG_ARRAYTYPE_P(0);

ExceptthatinsteadofreturninganINT32value,itreturnsanarraypointerARRAYTYPE_P.

Aftergettingthearraypointer,weperformacoupleofchecks:

Assert(ARR_ELEMTYPE(input_array)==INT4OID);

Weassertthattheelementtypeofreturnedarrayisindeedaninteger.(TherearesomeinconsistenciesinPostgreSQLcodeastheplainintegertypecanbecalledeitherint32orint4dependingonwherethedefinitioncomesfrom.Buttheybothmeanthesamething,exceptthatoneisbasedonthelengthinbitsandtheotherinbytes.)

Thetypecheckisanassertandnotaplainruntimecheck,becauseafteryouhaveyourSQLdefinitionpartofthefunctioninplace,PostgreSQLitselftakescarenottocallthefunctionwithanyothertypeofarray.

Thesecondcheckisforcheckingthattheargumentisreallyaone-dimensionalarray(PostgreSQLarrayscanhave1tondimensionsandstillbeofthesametype),asshown:


ereport(ERROR,


errmsg("useonlyone-dimensionalarrays!")));

Iftheinputarrayhasmorethanonedimension,weraiseanerror.(WewilldiscussPostgreSQL’serrorreportinginClater,initsownsection.)

Ifyouneedtoworkonarraysofanarbitrarynumberofdimensions,takealookatthesourcecodeoftheunnest()SQLfunctionwhichturnsanyarrayintoasetofarrayelements.

NoteThecodeislocatedinthebackend/utils/adt/arrayfuncs.cfileintheCfunctionarray_unnest(...).

Afterwehavedonebasicsanitycheckingontheargument,wearereadytostartprocessingthearray.AsaPostgreSQLarraycanbequiteacomplexbeast,withmultipledimensionsandarrayelementsstartingatanarbitraryindex,itiseasiesttouseaready-madeutilityfunctionformosttasks.So,hereweusethedeconstruct_array(...)functiontoextractaPostgreSQLarrayinthreeseparateCvariables:

Datum*datums;

bool*nulls;

intcount;

Thedatumspointerwillbesettopointtoanarrayfilledwithactualelements.The*nullspointerwillcontainapointertoanarrayofBooleans,whichwillbetrueifthecorrespondingarrayelementwasNULL,andcountwillbesettothenumberofelementsfoundinthearray,asshown:

deconstruct_array(input_array,//one-dimensionalarray

INT4OID,//ofintegers

4,//sizeofintegerinbytes

true,//int4ispass-byvalue

'i',//alignmenttypeis'i'

&datums,&nulls,&count);//resulthere

Theotherargumentsareasfollows:

input_array:ThisisthepointertothePostgreSQLarrayINT4OID:Thisisthetypeofarrayelementelementsize:Thisisthein-memorysizeoftheelementtypetrue:Thisistheelementpass-by-valueelementalignmentid

ThetypeOIDforint4(=23)isalreadyconvenientlydefinedasINT4OID.Theothers,youjusthavetolookup.

Theeasiestwaytogetthevaluesfortype,size,passbyvalue,andalignmentistoquerythesefromthedatabase:

c_samples=#selectoid,typlen,typbyval,typalignfrompg_type

c_samples-#wheretypname='int4';

-[RECORD1]

oid|23

typlen|4

typbyval|t

typalign|i

Afterthecalltodeconstruct_array(...)therestiseasy—justiterateoverthevalueandnullarraysandaccumulatethesum:

for(i=0;i<count;i++){

//firstcheckandignorenullelements

if(nulls[i])

continue;

//accumulateandremembertherewerenon-nullvalues

sum+=DatumGetInt32(datums[i]);

not_null=true;

}

TheonlyPostgreSQL-specificthinghereistheuseoftheDatumGetInt32(<datum>)macroforconvertingtheDatumtointeger.TheDatumGetInt32(<datum>)macroperformsnocheckingofitsargumenttoverifythatitisindeedaninteger(ifyouremember,thisisC,sonotypeofinfoisavailableinthedataitself),butusingtheDatumGet*()macrohelpsustomakethecompilerhappy.

Wearealmostdonehere,asreturningthesum(orNULLincaseallelementswereNULLvalues)isexactlythesameasinourpreviousfunction.

WhilethisisallfromtheCside,westillneedtoteachPostgreSQLaboutthisnewfunction.Thesimplestwayistodeclareafunctionwhichtakesanint[]argument:

CREATEORREPLACEFUNCTIONadd_arr(int[])RETURNSint

AS'$libdir/add_func','add_int32_array'

LANGUAGECSTRICT;

Itworksfineforanyintegerarrayyoupassitfor:

hannu=#SELECTadd_arr('{1,2,3,4,5,6,7,8,9}');

-[RECORD1]

add_arr|45

hannu=#SELECTadd_arr(ARRAY[1,2,NULL]);

-[RECORD1]

add_arr|3

hannu=#SELECTadd_arr(ARRAY[NULL::int]);

-[RECORD1]

add_arr|

Itevendetectsmultidimensionalarraysanderrorsoutifitispassedone:

hannu=#selectadd_arr('{{1,2,3},{4,5,6}}');

ERROR:1-dimensionalarrayneeded

Whatifwewanttouseitthesamewayasourtwo-argumentadd(a,b)function?

SinceVersion8.4ofPostgreSQL,itispossibleusingsupportforVARIADICfunctions,orfunctionstakingavariablenumberofarguments.

Createthefunctionasfollows:

CREATEORREPLACEFUNCTIONadd(VARIADICaint[])RETURNSint

AS'$libdir/add_func','add_int32_array'

LANGUAGECSTRICT;

Thepreviouscallstoadd_arr()canberewrittenas:

hannu=#selectadd(1,2,3,4,5,6,7,8,9);

-[RECORD1]

add|45

hannu=#selectadd(NULL);

-[RECORD1]

add|

hannu=#selectadd(1,2,NULL);

-[RECORD1]

add|3

Noticethatyoucan’teasilygetERROR:1-dimensionalarrayneededasVARIADICalwaysconstructsaone-dimensionalarrayfromthearguments.

Theonlythingmissingisthatyoucan’thavePostgreSQL’sfunctionoverloadingmechanismtodistinguishbetweenadd(aint[])andadd(VARIADICaint[])—yousimplycan’tdeclarebothoftheseatthesametimebecauseforPostgreSQLtheyarethesamefunctionwithonlytheinitialargumentdetectiondonedifferently.Thatiswhythearrayversionofthefunctionwasnamedadd_arr.IncaseyouneedtocalloneVARIADICfunctionfromanother,thereisaway.YoucancalltheVARIADICversionwithanargumentofthearraytypebyprefixingtheargumentwithVARIADIConthecallsideashannu=#selectadd(ARRAY[1,2,NULL]);:

ERROR:functionadd(integer[])doesnotexist

LINE1:selectadd(ARRAY[1,2,NULL]);

HINT:Nofunctionmatchesthegivennameandargumenttypes.Youmight

needtoaddexplicittypecasts.

hannu=#selectadd(VARIADICARRAY[1,2,NULL]);

-[RECORD1]

add|3

Youcanevensmuggleinamultidimensionalarray:

hannu=#selectadd(VARIADIC'{{1,2,3},{4,5,6}}');

ERROR:1-dimensionalarrayneeded

Thiscallingconventionalsomeans,thatevenwhenyoucreateVARIADICfunctions,youneedtocheckthearraydimensions.

BasicguidelinesforwritingCcodeAfterhavingwrittenourfirstfunction,let’slookatsomeofthebasiccodingguidelinesforPostgreSQLbackendcoding.

MemoryallocationOneoftheplacesyouhavetobeextracarefulwhenwritingCcodeingeneralismemorymanagement.Foranynon-trivialCprogram,youhavetocarefullydesignandimplementyourprograms,sothatallyourallocatedmemoryisfreedwhenyouaredonewithit,orelseyouwill“leakmemory”andwillprobablyrunoutofmemoryatsomepoint.

AsthisisalsoacommonconcernforPostgreSQL,ithasitsownsolution—memorycontexts.Let’stakeadeeperdiveintothem.

Usepalloc()andpfree()MostPostgreSQLmemoryallocationsaredoneusingPostgreSQL’smemoryallocationfunctionpalloc()andnotstandardCmalloc().Whatmakespalloc()specialisthatitallocatesthememoryinthecurrentcontextandthewholememoryisfreedinonegowhenthecontextisdestroyed.Forexample,thetransactioncontext—whichisthecurrentcontextwhenauser-definedfunctioniscalled—isdestroyedandmemoryallocatedisfreedattheendoftransaction.Thismeansthatmosttimestheprogrammersdonotneedtoworryabouttrackingpalloc()allocatedmemoryandfreeingit.

Itisalsoeasytocreateyourownmemorycontextsifyouhavesomememoryallocationneedswithdifferentlifespans.Forexample,thefunctionsforreturningasetofrows(describedinfurtherdetaillaterinthischapter)haveastructurepassedtothem,whereoneofthemembersisreservedforapointertoatemporarycontextspecificallyforkeepingafunction-levelmemorycontext.

Zero-fillthestructuresAlwaysmakesurethatnewstructuresarezero-filled,eitherbyusingmemsetafterallocatingthemorusingpalloc0().

PostgreSQLsometimesreliesonlogicallyequivalentdataitemsbeingalsothesameforbit-wisecomparisons.Evenwhenyousetalltheitemsinastructure,itispossiblethatsomealignmentissuesleavegarbageintheareasbetweenstructureelementsifanyalignmentpaddingwasdonebythecompiler.

Ifyoudon’tdothis,thenhashindexesandhashjoinsofPostgreSQLmayworkinefficientlyorevengivewrongresults.Theplanner’sconstantcomparisonsmayalsobewrongifconstantswhicharelogicallythesamearenotthesameviabit-wiseequality,resultinginundesirableplanningresults.

IncludefilesMostofPostgreSQLinternaltypesaredeclaredinpostgres.h,andthefunctionmanagerinterfaces(PG_MODULE_MAGIC,PG_FUNCTION_INFO_V1,PG_FUNCTION_ARGS,PG_GETARG_<type>,PG_RETURN_<type>,andsoon)areinfmgr.h.Therefore,allyourCextensionmodulesneedtoincludeatleastthesetwofiles.Itisagoodhabittoincludepostgres.hfirstasitgivesyourcodethebestportabilityby(re)definingsomeplatformdependentconstantsandmacros.Includingpostgres.halsoincludesutils/elog.handutils/palloc.hforyou.

Thereareotherusefulincludefilesintheutils/subdirectorywhichyoualsomayneedtoincludelikeutils/array.husedinthelastexample.

Anotheroftenusedincludedirectoryiscatalog/whichgivesyoutheinitial(andbyconventionconstant)partofmostsystemtables,soyoudonotneedtolookupthingsliketypeidentifierfortheint4datatype,butcanuseitspredefinedvalueINT4OIDdirectly.

Thevaluesincatalog/pg_*includefilesarealwaysinsyncwithwhatgetsputintothedatabasecatalogsbyvirtueofbeingthedefinitionofthestructureandcontentsofthesystemcatalogtables.The.bkifilesusedwhentheinitdbcommandsetsupanewemptydatabaseclusteraregeneratedfromthese.hfilesbythegenbki.plscript.

PublicsymbolnamesItistheprogrammer’stasktomakesurethatanysymbolnamesvisibleinthe.sofilesdonotconflictwiththosealreadypresentinthePostgreSQLbackend,includingthoseusedbyotherdynamicallyloadedlibraries.Youwillhavetorenameyourfunctionsorvariablesifyougetmessagestothiseffect.Picksufficientlydistinctivenamesforfunctionstoavoidanysuchproblems.Avoidshortnamesthatmightbeasourceofpotentialconflict.Thismaybeabiggerproblemiftheconflictscomefromathird-partylibraryyourcodeisusing.SotestearlyinthedevelopmentifyoucanlinkalltheplannedlibrariestoyourPostgreSQLextensionmodule.

ErrorreportingfromCfunctionsOnethingwhichwentunexplainedintheprevioussamplewastheerrorreportingpart:


ereport(ERROR,


errmsg("useonlyone-dimensionalarrays!")));

Allerrorreportingandotheroff-channelmessaginginPostgreSQLisdoneusingtheereport(<errorlevel>,rest)macrowhosemainpurposeistomakeerrorreportinglooklikeafunctioncall.

Theonlyparameterwhichisprocesseddirectlybyereport()isthefirstargumenterrorlevel,orperhapsmoreexactly,theseveritylevelorloglevel.Alltheotherparametersareactuallyfunctioncallswhichindependentlygenerateandstoreadditionalerrorinformationinthesystemtobewrittentologsand/orbesenttotheclient.Beingplacedintheargumentlistoftheereport()makessurethattheseotherfunctionsarecalledbeforetheactualerrorisreported.ThisisimportantbecauseinthecaseofanerrorlevelbeingERROR,FATAL,orPANICthesystemcleansupallcurrenttransactionstatesandanythingaftertheereport()callwillnevergetachancetorun.Errorstatestheendofthetransaction.

IncaseofERROR,thesystemisreturnedtoacleanstateanditwillbereadytoacceptnewcommands.

ErrorlevelFATALwillcleanupthebackendandexitthecurrentsession.

PANICisthemostdestructiveoneanditwillnotonlyendthecurrentconnection,butwillalsocauseallotherconnectionstobeterminated.PANICmeansthatsharedstate(sharedmemory)ispotentiallycorruptedanditisnotsafetocontinue.Itisusedautomaticallyforthingslikecoredumpsorother“hard”crashes.

“Error”statesthatarenoterrorsWARNINGisthehighestnon-errorlevel.Itmeansthatsomethingmaybewrongandneedsuser/administratorattention.Itisagoodpracticetoperiodicallyscansystemlogsforwarnings.Usethisonlyforunexpectedconditions.Seethenextoneforthingshappeningonaregularbasis.Warningsgotoclientandserverlogsbydefault.

NOTICEisforthingswhicharelikelyofhigherinteresttousers,likeinformationaboutcreatingaprimarykeyindexorsequenceforserialtype(thoughthesestoppedtobeNOTICEinthelatestversionofPostgreSQL).Likethepreviousone,NOTICEissentbothtoclientandserverlogsbydefault.

INFOisforthingsspecificallyrequestedbyclient,likeVACUUM/ANALYSEVERBOSE.Itisalwayssenttotheclientregardlessoftheclient_min_messagesGUCsetting,butisnotwrittentoaserverlogwhenusingdefaultsettings.

LOGandCOMMERRORareforservers,operationalmessages,andbydefaultareonlywrittentotheserverlog.TheerrorlevelLOGcanalsobesenttotheclientifclient_min_messagesissetappropriately,butCOMMERRORneveris.

ThereareDEBUG1toDEBUG5inincreasingorderofverbosity.Theyarespecificallymeantforreportingdebugginginfoandarenotreallyusefulinmostothercases,exceptperhapsforcuriosity.SettinghigherDEBUGxlevelsisnotrecommendedinproductionservers,astheamountloggedorreportedcanbereallyhuge.

Whenaremessagessenttotheclient?Whilemostcommunicationfromtheservertotheclienttakesplaceafterthecommandcompletes(orevenafterthetransactioniscommittedincaseofLISTEN/NOTIFY),everythingemittedbyereport()issenttotheclientimmediately,thusthementionofoff-channelmessagingpreviously.Thismakesereport()ausefultoolformonitoringlong-runningcommandssuchasVACUUMandalsoasimpledebuggingaidtoprintoutusefuldebuginfo.

NoteYoucanreadamuchmoredetaileddescriptionoferrorreportingathttp://www.postgresql.org/docs/current/static/error-message-reporting.html.

http://www.postgresql.org/docs/current/static/error-message-reporting.html

RunningqueriesandcallingPostgreSQLfunctionsOurnextstopisrunningSQLqueriesinsidethedatabase.Whenyouwanttorunaqueryagainstthedatabase,youneedtousesomethingcalledServerProgrammingInterface(SPI).SPIgivesprogrammertheabilitytorunSQLqueriesviaasetofinterfacefunctionsforusingPostgreSQL’sparser,planner,andexecutor.

NoteIftheSQLstatementyouarerunningviaSPIfails,thecontrolisnotreturnedtothecaller,butinsteadthesystemrevertstoacleanstateviainternalmechanismsforROLLBACK.ItispossibletocatchSQLerrorsbyestablishingasub-transactionaroundyourcalls.Itisaninvolvedprocessnotyetofficiallydeclared“stable”andtherefore,itisnotpresentinthedocumentationonCextensions.Ifyouneedit,onegoodplacetolookatwouldbethesourcecodeforvariouspluggablelanguages(PL/python,PL/proxy,andsoon)whichdoitandarelikelytobemaintainedingoodorderiftheinterfacechanges.

InthePL/pythonsource,thefunctionstoexamineareintheplpython/plpy_spi.cfileandareappropriatelynamedPly_spi_subtransaction_[begin|commit|abort]().

TheSPIfunctionsdoreturnnon-negativevaluesforsuccess,eitherdirectlyviaareturnvalueorinaglobalvariableSPI_result.ErrorsproduceanegativevalueorNull.

AsampleCfunctionusingSPIHereisasamplefunctiondoinganSQLqueryviaSPI_*()functions.Itisamodifiedversionofthesamplefromthestandarddocumentation(itusesVersion1callingconventionsandoutputsanextrabitofinformation).The.c,.sql.in,andMakefilefunctionsforthissampleareavailableinthespi_samples/subdirectory.

Datum

count_returned_rows(PG_FUNCTION_ARGS)

{

char*command;

intcnt;

intret;

intproc;

/*getarguments,convertcommandtoCstring*/

command=text_to_cstring(PG_GETARG_TEXT_P(0));

cnt=PG_GETARG_INT32(1);

/*openinternalconnection*/

SPI_connect();

/*runtheSQLcommand*/

ret=SPI_exec(command,cnt);

/*savethenumberofrows*/

proc=SPI_processed;

/*Ifsomerowswerefetched,printthemviaelog(INFO).*/

if(ret>0&&SPI_tuptable!=NULL)

{

TupleDesctupdesc=SPI_tuptable->tupdesc;

SPITupleTable*tuptable=SPI_tuptable;

charbuf[8192];

inti,j;

for(j=0;j<proc;j++)

{

HeapTupletuple=tuptable->vals[j];

//constructastringrepresentingthetuple

for(i=1,buf[0]=0;i<=tupdesc->natts;i++)

snprintf(buf+strlen(buf),

sizeof(buf)–strlen(buf),

"%s(%s::%s)%s",

SPI_fname(tupdesc,i),

SPI_getvalue(tuple,tupdesc,i),

SPI_gettype(tupdesc,i),

(i==tupdesc->natts)?"":"|");

ereport(INFO,(errmsg("ROW:%s",buf)));

}

}

SPI_finish();

pfree(command);

PG_RETURN_INT32(proc);

}

AftergettingtheargumentsusingthePG_GETARG_*macro,thefirstnewthingshownisopeninganinternalconnectionviaSPI_connect()whichsetsuptheinternalstateforthefollowingSPI_*()functioncalls.ThenextstepistoexecuteafullSQLstatementusingSPI_exec(command,cnt).

TheSPI_exec()functionisaconvenientvariantofSPI_execute(...)withtheread_onlyflagsettofalse.Theread_onlyflag,ifsettotruemeansthatonlyareadonlycommandcanbeexecuted.ThereisalsoathirdversionoftheexecuteatonceSPIfunction,theSPI_execute_with_args(...),whichpreparesthequery,bindsthepassed-inarguments,andexecutesinasinglecall.

Afterthequeryisexecuted,wesavetheSPI_processedvalueforreturningthenumberofrowsprocessedattheendofthefunction.Inthissample,itisnotstrictlynecessary,butingeneralyouneedtosaveanySPI_*globalvariablebecausetheycouldbeoverwrittenbythenextSPI_*(...)call.

ToshowwhatwasreturnedbythequeryandalsotoshowhowtoaccessfieldsreturnedbySPIfunctions,wenextprintoutdetailedinfoonanytuplesreturnedbythequeryviatheereport(INFO,…)call.WefirstcheckedthattheSPI_execcallwassuccessful(ret>0)andthatsometupleswerereturned(SPI_tuptable!=NULL),andthenforeachreturnedtuplefor(j=0;j<proc;...)weloopedoverthefieldsfor(i=1;i<=tupdesc->natts;...)formattingthefieldsinfointoabuffer.Wegetthestringrepresentationsofthefieldname,value,anddatatypeusingSPIfunctionsSPI_fname(),SPI_getvalue(),andSPI_gettype()andthensendtherowtotheuserusingereport(INFO,…).Ifyouwanttoreturnthevaluesfromthefunctioninstead,seethenextsectionsonreturningtheSETOFvaluesandcompositetypes.

Finally,wefreedtheSPIinternalstateusingSPI_finish();.Onecanalsofreethespaceallocatedforthecommandvariablebythetext_to_cstring(<textarg>)function,thoughitisnotstrictlynecessary,thankstothefunctioncallcontextbeingdestroyedandmemoryallocatedinitbeingfreedanywayatthefunctionexit.

VisibilityofdatachangesThevisibilityrulesfordatachangesinPostgreSQLarethateachcommandcannotseeitsownchangesbutusuallycanseechangesmadebycommandswhichwerestartedbeforeit,evenwhenthecommandisstartedbytheoutercommandorquery.

Theexceptioniswhenthequeryisexecutedwitharead-onlyflagset,inwhichcasethechangesmadebyoutercommandsareinvisibletoinnerorcalledcommands.

Thevisibilityrulesaredescribedinthedocumentationathttp://www.postgresql.org/docs/current/static/spi-visibility.htmlandmaybequitecomplextounderstandatfirst,butitmayhelptothinkofaread-onlySPI_execute()callasbeingcommand-level,similartothetransactionisolationlevelserializable;andaread-writecallsimilartotheread-committedisolationlevel.

Thisisfurtherexplainedathttp://www.postgresql.org/docs/current/static/spi-examples.htmlintheSamplesessionsection.

http://www.postgresql.org/docs/current/static/spi-visibility.html

http://www.postgresql.org/docs/current/static/spi-examples.html

MoreinfoonSPI_*functionsThereisalotmoreinformationonspecificSPI_*()functionsintheofficialdocumentation.

ForPostgreSQLVersion9.3functions,http://www.postgresql.org/docs/current/static/spi.htmlisthestartingpointfortheSPIdocs.

MoresamplecodeisalsoavailableinthePostgreSQLsourceinregressiontestsatsrc/test/regress/regress.candinthecontrib/spi/module.

http://www.postgresql.org/docs/current/static/spi.html

HandlingrecordsasargumentsorreturnedvaluesAsournextexercise,let’swriteafunctionwhichtakesarecordofthreeintegersa,b,andcasanargumentandreturnsasetofdifferentrecords—allpermutationsofa,b,andcwithanextrafieldxcomputedasa*b+c.

First,thisfunctioniswritteninPL/pythontomakeiteasiertounderstandwhatwearetryingtodo:

hannu=#CREATELANGUAGEplpythonu;

CREATELANGUAGE

hannu=#CREATETYPEabcAS(aint,bint,cint);

CREATETYPE

hannu=#CREATEORREPLACEFUNCTION

hannu-#reverse_permutations(rabc)

hannu-#RETURNSTABLE(cint,bint,aint,xint)

hannu-#AS$$

hannu$#a,b,c=r['a'],r['b'],r['c']

hannu$#yielda,b,c,a*b+c

hannu$#yielda,c,b,a*c+b

hannu$#yieldb,a,c,b*b+c

hannu$#yieldb,c,a,b*c+a

hannu$#yieldc,a,b,c*a+b

hannu$#yieldc,b,a,c*b+a

hannu$#$$LANGUAGEplpythonu;

CREATEFUNCTION

hannu=#SELECT*FROMreverse_permutations(row(2,7,13));

-[RECORD1]

c|2

b|7

a|13

x|27

-[RECORD2]

c|2

b|13

a|7

x|33

-[RECORD3]

c|7

b|2

a|13

x|62

-[RECORD4]

c|7

b|13

a|2

x|93

-[RECORD5]

c|13

b|2

a|7

x|33

-[RECORD6]

c|13

b|7

a|2

x|93

TherearethreenewthingsthatwearegoingtotouchinthefollowingCimplementationofasimilarfunction:

HowtofetchanelementofRECORDpassedasanargumentHowtoconstructatupletoreturnaRECORDtypeHowtoreturnSETOF(alsoknownasTABLE)ofthisRECORD

Solet’sdiveintotheCcodeforthisrightaway(asamplecanbefoundinthechap9/c_records/directory).

Forbetterclarity,wewillexplainthisfunctionintwoparts:first,doingasimplereverse(a,b,c)function,whichreturnsjustasinglerecordof(c,b,a,x=c*b+a),andthenexpandingittoreturnasetofpermutationssuchasthesamplePL/pythonufunction.

ReturningasingletupleofacomplextypeThefirststepinconstructingaversionofthereversepermutationsfunctioninCistostartwithsimplybeingabletoreturnasinglerecordoftypeabc:

Datum

c_reverse_tuple(PG_FUNCTION_ARGS)

{

HeapTupleHeaderth;

int32a,b,c;

boolaisnull,bisnull,cisnull;

TupleDescresultTupleDesc;

OidresultTypeId;

Datumretvals[4];

boolretnulls[4];

HeapTuplerettuple;

//getthetupleheaderof1stargument

th=PG_GETARG_HEAPTUPLEHEADER(0);

//getargumentDatum'sandconvertthemtoint32

a=DatumGetInt32(GetAttributeByName(th,"a",&aisnull));

b=DatumGetInt32(GetAttributeByName(th,"b",&bisnull));

c=DatumGetInt32(GetAttributeByName(th,"c",&cisnull));

//debug:reporttheextractedfieldvalues

ereport(INFO,

(errmsg("arg:(a:%d,b:%d,c:%d)",a,b,c)));

//setuptupledescriptorforresultinfo

get_call_result_type(fcinfo,&resultTypeId,&resultTupleDesc);

//checkthatSQLfunctiondefinitionissetuptoreturnarecord

Assert(resultTypeId==TYPEFUNC_COMPOSITE);

//makethetupledescriptorknowntopostgresasvalidreturntype

BlessTupleDesc(resultTupleDesc);

retvals[0]=Int32GetDatum(c);

retvals[1]=Int32GetDatum(b);

retvals[2]=Int32GetDatum(a);

retvals[3]=Int32GetDatum(retvals[0]*retvals[1]+retvals[2]);

retnulls[0]=aisnull;

retnulls[1]=bisnull;

retnulls[2]=cisnull;

retnulls[3]=aisnull||bisnull||cisnull;

rettuple=heap_form_tuple(resultTupleDesc,retvals,retnulls);

PG_RETURN_DATUM(HeapTupleGetDatum(rettuple));

}

ExtractingfieldsfromanargumenttupleGettingthefieldsofanargumenttupleiseasy.First,youfetchtheHeapTupleHeaderfileoftheargumentintothethvariableusingthePG_GETARG_HEAPTUPLEHEADER(0)macro,andthenforeachfieldyougettheDatum(agenerictypewhichcanholdanyfieldvalueinPostgreSQL)bythefieldnameusingtheGetAttributeByName()functionandthenassignitsvaluetoalocalvariableafterconvertingittoint32viaDatumGetInt32():

a=DatumGetInt32(GetAttributeByName(th,"a",&aisnull));

ThethirdargumenttoGetAttributeByName(...)isanaddressofaboolwhichissettotrueifthefieldwasNULL.

ThereisalsoacompanionfunctionGetAttributeByNum()ifyouprefertogettheattributesbytheirnumbersinsteadofnames.

ConstructingareturntupleConstructingthereturntuple(s)isalmostaseasy.

First,yougetthecalledfunctionsreturntypedescriptorusingtheget_call_result_type()function:

get_call_result_type(fcinfo,&resultTypeId,&resultTupleDesc);

ThefirstargumenttothisfunctionistheFunctionCallInfostructurefcinfowhichisusedwhencallingthefunctionyouarecurrentlywriting(hiddenbehindthePG_FUNCTION_ARGSmacrointheCfunctiondeclaration).TheothertwoargumentsareaddressesofthereturntypeOidandTupleDesctoreceivethereturntupledescriptorincasethefunctionreturnsarecordtype.

Next,thereisasafetyassertforcheckingthatthereturntypeisreallyarecord(orcomposite)type:

Assert(resultTypeId==TYPEFUNC_COMPOSITE);

ThisistoguardagainsterrorsintheCREATEFUNCTIONdeclarationinSQLwhichtellsPostgreSQLaboutthisnewfunction.

Andtherestillremainsonethingbeforeweconstructthetuple:

BlessTupleDesc(resultTupleDesc);

ThepurposeofBlessTupleDesc()istofillinthemissingpartsofthestructure,whicharenotneededforinternaloperationsonthetuple,butareessentialwhenthetupleisreturnedfromthefunction.

Sowearedonewiththetupledescriptorandfinally,wecanconstructthetupleorrecordittobereturned.

Thetupleisconstructedusingtheheap_form_tuple(resultTupleDesc,retvals,retnulls);functionwhichusesTupleDescwejustprepared.ItalsoneedsanarrayofDatumtobeusedasvaluesinthereturntuple,andanarrayofbool,whichisusedtodetermineifanyfieldshouldbesettoNULLinsteadoftheircorrespondingDatumvalue.Asallourfieldsareoftypeint32,theirvaluesinretvalsaresetusingInt32GetDatum(<localvar>).Thearrayretnullisasimplearrayofboolandneedsnospecialtrickstosetitsvalues.

Finallywereturntheconstructedtuple:

PG_RETURN_DATUM(HeapTupleGetDatum(rettuple));

Here,wefirstconstructDatumfromthetuplewejustconstructedusingHeapTupleGetDatum()andthenusethePG_RETURN_DATUMmacro.

Interlude–whatisDatum?Inthischapter,weusesomethingcalledDatuminseveralplaces.ThiscallsforabitofexplanationaboutwhataDatumis.

Inshort,aDatumisanydataitemthePostgreSQLprocessesandpassesaround.DatumitselfdoesnotcontainanytypeinformationorinfoonwhetherthefieldisactuallyNULL.Itisjustapointertoamemory.Youalwayshavetofindout(orknowbeforehand)thetypeofanyDatumyouuseandalsohowtofindoutifyourdatamaybeNULLinsteadofanyrealvalue.

Intheprecedingexample,GetAttributeByName(th,"b",&bisnull)returnsaDatum,anditcanreturnsomethingevenwhenthefieldinthetupleisNULL,soalwayscheckfornull-nessfirst.Also,thereturnedDatumitselfcannotbeusedformuchunlessweconvertittosomerealtype,asdoneinthenextstepusingDatumGetInt32(),whichsimplyconvertsthevagueDatumtoarealint32valuebasicallydoingacastfromamemorylocationofanundefinedtypetoint32.

ThedefinitionofDatuminpostgresql.histypedefDatum*DatumPtr;thatisanythingpointedtobyaDatumPtr.EventhoughDatumPtrisdefinedastypedefuintptr_tDatum;itmaybeeasiertothinkofitasa(slightlyrestricted)void*.

Oncemore,anyrealsubstanceisaddedtoaDatumbyconvertingittoarealtype.

Youcanalsogotheotherway,turningalmostanythingintoaDatumasseenattheendofthefunction:

HeapTupleGetDatum(rettuple)

Again,foranythingelseinPostgreSQLtomakeuseofsuchDatum,thetypeinformationmustbeavailablesomewhereelse,inourcasethereturntypedefinitionsofthefunction.

ReturningasetofrecordsNext,wewillmodifyourfunctiontonotjustreturnasinglerecordofreorderedfieldsfromanargumentrecord,buttoreturnallpossibleorderings.Wewillstilladdoneextrafieldxasanexampleofhowyoucanusethevaluesyouextractedfromtheargument.

Forset-returningfunctions,PostgreSQLhasaspecialcallingmechanismwherePostgreSQL’sexecutormachinerywillkeepcallingthefunctionoverandoveragainuntilitreportsbackthatitdoesnothaveanymorevaluestoreturn.Thisreturn-and-continuebehaviorisverysimilartohowtheyieldkeywordworksinPythonorJavaScript.

Allcallstothesetreturningfunctiongetanargumentapersistentstructuremaintainedoutsidethefunctionandmadeavailabletothefunctionviamacros:SRF_FIRSTCALL_INIT()forthefirstcallandSRF_PERCALL_SETUP()forsubsequentcalls.

Tofurtherclarifytheexample,weprovideaconstantarrayofpossibleorderingstobeusedwhenpermutingthevalues.

Also,wereadargumentfieldsa,b,andconlyonceatthebeginningofthefunctionandsavetheextractedvaluesinastructurec_reverse_tuple_args,whichweallocateandinitializeatthefirstcall.Forthestructuretosurvivethroughallcalls,weallocatethisstructureinaspecialmemorycontextwhichismaintainedinthefuncctx->multi_call_memory_ctxandstorethepointertothisstructureinfuncctx->user_fctx.Wealsomakeuseoffuncctxfields:call_cntrandmax_calls.

Inthesamecodesectionrunonceatthefirstcall,wealsopreparethedescriptorstructureneededforreturningthetuples.Todoso,wefetchthereturntupledescriptorbypassingtheaddresswegetinfuncctx->tuple_desctothefunctionget_call_result_type(...),andwecompletethepreparationbycallingBlessTuple(...)onittofillinthemissingbitsneededforusingitforreturningvalues.

Attheendofthissection,werestorethememorycontext.Whileyouusuallydonotneedtopfree()thethingsyouhavepalloc()allocated,youshouldalwaysremembertorestorethememorycontextwhenyouaredoneusinganycontextyouhaveswitchedto,orelseyouriskmessingupPostgreSQLinawaythatcanbehardtodebuglater!

Therestissomethingthatgetsdoneateachcall,includingthefirstone.

Westartbycheckingthatthereisstillsomethingtodobycomparingthecurrentcalltothemaxcallsparameter.Thisisbynomeanstheonlywaytodetermineifwehavereturnedallvalues,butitisindeedthesimplestwayifyouknowaheadhowmanyrowsyouaregoingtoreturn.Iftherearenomorerowstoreturn,wesignalthisbackusingSRF_RETURN_DONE().

Therestisverysimilartowhattheprevioussingle-tuplefunctiondid.Wecomputetheretvalsandretnullsarraysusingtheindexpermutationsarrayipsandthenconstructatupletoreturnusingheap_form_tuple(funcctx->tuple_desc,retvals,retnulls);.

Finally,wereturnthetupleusingmacroSRF_RETURN_NEXT(...),convertingthetupletoDatum,asthisiswhatthemacroexpects.

Onemorethingtonote,allcurrentversionsofPostgreSQLwillalwayskeepcallingyourfunctionuntilitreturnsSRF_RETURN_DONE().Thereiscurrentlynowaytodoan“earlyexit”fromthecallersside.Thismeansthatifyourfunctionreturns1millionrowsandyoudo:

SELECT*FROMmymillionrowfunction()LIMIT3;

Thefunctionwillbecalled1milliontimesinternally,andalltheresultswillbecached,andonlyafterthis,thefirstthreerowswillbereturnedandtheremaining9,99,997rowsarediscarded.Thisisnotafundamentallimitation,butjustanimplementationdetailwhichislikelytochangeinsomefutureversionofPostgreSQL.Don’tholdyourbreaththough,thiswillonlyhappenifsomebodyfindsthisvaluableenoughtoimplement.

Thesourcewithmodificationsdescribedpreviouslyisasfollows:

structc_reverse_tuple_args{

int32argvals[3];

boolargnulls[3];

boolanyargnull;

};

Datum

c_permutations_x(PG_FUNCTION_ARGS)

{

FuncCallContext*funcctx;

constchar*argnames[3]={"a","b","c"};

//6possibleindexpermutationsfor0,1,2

constintips[6][3]={{0,1,2},{0,2,1},

{1,0,2},{1,2,0},

{2,0,1},{2,1,0}};

inti,call_nr;

structc_reverse_tuple_args*args;

if(SRF_IS_FIRSTCALL())

{

HeapTupleHeaderth=PG_GETARG_HEAPTUPLEHEADER(0);

MemoryContextoldcontext;

/*createafunctioncontextforcross-callpersistence*/

funcctx=SRF_FIRSTCALL_INIT();

/*switchtomemorycontextappropriateformultiplefunctioncalls

*/

oldcontext=MemoryContextSwitchTo(

funcctx->multi_call_memory_ctx

);

/*allocateandzero-fillstructforpersistingextracted

arguments*/

args=palloc0(sizeof(structc_reverse_tuple_args));

args->anyargnull=false;

funcctx->user_fctx=args;

/*totalnumberoftuplestobereturned*/

funcctx->max_calls=6;

//thereare6permutationsof3elements

//extractargumentvaluesandNULL-ness

for(i=0;i<3;i++){

args->argvals[i]=DatumGetInt32(GetAttributeByName(th,

argnames[i],&(args->argnulls[i])));

if(args->argnulls[i])

args->anyargnull=true;

}

//setuptupleforresultinfo

if(get_call_result_type(fcinfo,NULL,&funcctx->tuple_desc)

!=TYPEFUNC_COMPOSITE)

ereport(ERROR,

(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),

errmsg("functionreturningrecordcalledincontext"

"thatcannotaccepttyperecord")));

BlessTupleDesc(funcctx->tuple_desc);

//restorememorycontext

MemoryContextSwitchTo(oldcontext);

}

funcctx=SRF_PERCALL_SETUP();

args=funcctx->user_fctx;

call_nr=funcctx->call_cntr;

if(call_nr<funcctx->max_calls){

HeapTuplerettuple;

Datumretvals[4];

boolretnulls[4];

for(i=0;i<3;i++){

retvals[i]=Int32GetDatum(args->argvals[ips[call_nr][i]]);

retnulls[i]=args->argnulls[ips[call_nr][i]];

}

retvals[3]=Int32GetDatum(args->argvals[ips[call_nr][0]]

*args->argvals[ips[call_nr][1]]

+args->argvals[ips[call_nr][2]]);

retnulls[3]=args->anyargnull;

rettuple=heap_form_tuple(funcctx->tuple_desc,retvals,retnulls);

SRF_RETURN_NEXT(funcctx,HeapTupleGetDatum(rettuple));

}

else/*dowhenthereisnomoreleft*/

{

SRF_RETURN_DONE(funcctx);

}

}

FastcapturingofdatabasechangesSomeobviousthingstocodeinCareloggingorauditingtriggers,whichgetcalledateachINSERT,UPDATE,orDELETEtoatable.WehavenotsetasideenoughspaceinthisbooktoexplaineverythingneededforCtriggers,butinterestedreaderscanlookupthesourcecodefortheskytoolspackagewhereyoucanfindmorethanonewaytowritetriggersinC.

ThehighlyoptimizedCsourceforthetwomaintriggers:logtrigaandlogutriga,includeseverythingyouneedtocapturethesechangestoatableandalsotodetecttablestructurechangeswhilethecodeisrunning.

Thelatestsourcecodeforskytoolscanbefoundathttp://pgfoundry.org/projects/skytools.

http://pgfoundry.org/projects/skytools

Doingsomethingatcommit/rollbackAsofthiswriting,thereisnopossibilitytodefineatriggerfunctionwhichisexecutedONCOMMITorONROLLBACK.However,ifyoureallyneedtohavesomecodeexecutedonthesedatabaseevents,youhaveapossibilitytoregisteraC-languagefunctiontobecalledontheseevents.Unfortunately,thisregistrationcannotbedoneinapermanentwayliketriggers,buttheregistrationfunctionhastobecalledeachtimeanewconnectionstarts:

RegisterXactCallback(my_xact_callback,NULL);

Usegrep-rRegisterXactCallbackinthecontrib/directoryofPostgreSQL’ssourcecodetofindfileswithexamplesofactualcallbackfunctions.

SynchronizingbetweenbackendsAlltheprecedingfunctionsaredesignedtoruninasingleprocess/backendasiftheotherPostgreSQLprocessesdidnotexist.

Butwhatifyouwanttologsomethingtoasinglefilefrommultiplebackends?

Seemseasy—justopenthefileandwritewhatyouwant.Unfortunately,itisnotthateasyifyouwanttodoitfrommultipleparallelprocessesandyoudonotoverwriteormixupthedatawithwhatotherprocesseswrite.

Tohavemorecontroloverthewritingorderbetweenbackends,youneedtohavesomekindofinter-processsynchronization,andtheeasiestwaytodothisinPostgreSQListousesharedmemoryandlight-weightlocks(LWLocks).

Toallocateitsownsharedmemorysegmentyour.sofileneedstobepreloaded,thatis,itshouldbeoneofthepreloadedlibrariesgiveninthepostgresql.confvariableshared_preload_libraries.

Inthe_PG_init()functionofyourmodule,youaskfortheaddressofanamesharedmemorysegment.Ifyouarethefirstoneaskingforthesegment,youarealsoresponsibleforinitializingthesharedstructures;including,creating,andstoringanyLWLocksyouwishtouseinyourmodule.

WritingfunctionsinC++ItisabadideatomixPostgreSQLwithC++foranumberofreasons.ItisbettertowrapyourC++codeintoCcodebehindexternCfunctions.Thiscanbeaproblemifyouheavilyusetemplatesandlibrarieslikeboost.

FormorediscussiononuseofC++readthePostgreSQLdocumentationherehttp://www.postgresql.org/docs/current/static/xfunc-c.html#EXTEND-CPP.

http://www.postgresql.org/docs/current/static/xfunc-c.html#EXTEND-CPP

AdditionalresourcesforCInthischapter,wewereabletoonlygiveyouaverybasicintroductiontowhatispossibleinC.Hereissomeadviceonhowtogetmoreinformation.

First,thereisofcoursethechapterC-LanguageFunctionsinthePostgreSQLmanual.Thiscanbefoundonlineathttp://www.postgresql.org/docs/current/static/xfunc-c.htmlandaswithmostoftheonlinePostgreSQLmanuals,youusuallycangettoolderversionsiftheyexist.

Thenextone,surprisingly,isthePostgreSQLsourcecodeitself.However,youwillusuallynotgetveryfarbyjustopeningthefilesorusinggreptofindwhatyouneed.Ifyouaregoodwithusingctags(http://en.wikipedia.org/wiki/Ctags)oranothersimilartool,itisdefinitelyrecommended.

Also,ifyouarenewtothesetypesoflarge-codeexplorationsystems,thenareallygoodresourceforfindingandexaminingPostgreSQLinternalsismaintainedathttp://doxygen.postgresql.org/.Thispointstothelatestgitmaster,soitmaynotbeaccurateforyourversionofPostgreSQL,butitisusuallygoodenoughandatleastprovidesanicestartingpointfordiggingaroundinthesourcecodeofyourversion.

Quiteoften,youwillfindsomethingtobase(partsof)yourCsourceoninthecontrib/directoryinthesourcecode.Togetanideaofwhatliesthere,readthroughtheAppendixF,AdditionalSuppliedModules(http://www.postgresql.org/docs/current/static/contrib.html).It’sentirelypossiblethatsomebodyhasalreadywrittenwhatyouneed.Therearemanymoremodulesinhttp://pgfoundry.orgforyoutoexamineandchoosefrom.Awordofwarningthough,whilemodulesincontrib/arecheckedatleastbyoneortwocompetentPostgreSQLcoreprogrammers,thethingsatpgfoundrycanbeofwildlyvaryingquality.Thetopactiveprojectsarereallygood;however,sothemainthingstolookatwhendeterminingifyoucanusethemasalearningsourcearehowactivetheprojectisandwhenwasitlastupdated.

ThereisalsoasetofGUCparametersspecificallyfordevelopmentanddebuggingwhichareusuallyleftoutofthesamplepostgresql.conffile.Thedescriptionsandanexplanationareavailableathttp://www.postgresql.org/docs/current/static/runtime-config-developer.html.

http://www.postgresql.org/docs/current/static/xfunc-c.html

http://en.wikipedia.org/wiki/Ctags

http://doxygen.postgresql.org/

http://www.postgresql.org/docs/current/static/contrib.html

http://pgfoundry.org

http://www.postgresql.org/docs/current/static/runtime-config-developer.html

SummaryAsCisthelanguagethatPostgreSQLitselfiswrittenin,itisveryhardtodrawadistinctiononwhatisanextensionfunctionusingadefinedAPIandwhatishackingPostgreSQLitself.

Someofthetopicsthatwedidnottouchatallwere:

Creatingnewinstallabletypesfromscratch—seecontrib/hstore/forafullimplementationofanewtype.Creatingnewindexmethods—downloadanolderversionofPostgreSQLtoseehowfulltextindexingsupportwasprovidedasanadd-on.ImplementinganewPL/*language—searchforpl/lolcodeforalanguage,thesolepurposeofwhichistodemonstratehowaPostgreSQL’sPL/*languageshouldbewritten(seehttp://pgfoundry.org/projects/pllolcode/).YoumayalsowanttocheckoutthesourcecodeforPL/Proxyforacleanandwell-maintainedPLlanguage.(TheusageofPL/Proxyisdescribedinthenextchapter.)

Hopefully,thischaptergaveyouenoughinfotoatleaststartwritingPostgreSQLextensionfunctionsinC.

IfyouneedmorethanwhatisavailablehereorintheofficialPostgreSQLdocumentation,thenrememberthatlotsofPostgreSQL’sbackenddeveloperdocumentation—oftenincludinganswerstothequestionsHow?andWhy?—isinthesourcefiles.AlotofthatcanalsoberelevanttoCextensions.

Soremember—UseTheSource,Luke!

InthenextchapterwewilldiscussscalingthedatabaseusingPL/Proxy.

http://pgfoundry.org/projects/pllolcode/

Chapter10.ScalingYourDatabasewithPL/ProxyIfyouhavefollowedtheadviceinthepreviouschaptersfordoingallyourdatabaseaccessthroughfunctions,youareinagreatpositiontoscaleyourdatabasebyhorizontallydistributingthedataovermultipleservers,alsoknownasdatabasesharding.Horizontaldistributionmeansthatyoukeepjustaportionofatableoneachpartitionofthedatabase,andthatyouhaveamethodtoautomaticallyaccesstherightdatabasewhenaccessingthedata.

WewillgentlyintroducetheconceptsleadingtothePL/Proxypartitioninglanguage,andthendelveintothesyntaxandproperusageofthelanguageitself.Let’sstartwithwritingascalableapplicationfromscratch.First,wewillwriteittobeashighlyperformingaspossibleononeserver.Then,wewillscaleitbyspreadingitoutonseveralservers.WewillfirstgetthisimplementedinPL/PythonuandtheninPL/Proxy.

NoteThisapproachisworthtaking,onlyifyouhave(plansfor)areallylargedatabase.Formostdatabases,oneserverplusoneorperhapstwohotstandbyserversshouldbemorethanenough.

Creatingasimplesingle-serverchatPerhaps,thesimplestapplicationneedingthiskindofscalabilityisamessaging(orchat)application;so,let’swriteone.

Theinitialsingle-serverimplementationhasthefollowingspecifications:

ThereshouldbeusersandmessagesEachuserhasausername,password,e-mail,listoffriends,andaflagtoindicateiftheuserwantstogetmessagesfromonlytheirfriends,orfromeverybodyForusers,therearemethodsfor:

RegisteringnewusersUpdatingthelistoffriendsLoggingin

Eachmessagehasasender,receiver,messagebody,andtimestampsforsendingandreadingthemessageFormessages,therearemethodsfor:

SendingamessageRetrievingnewmessages

Aminimalisticsystemimplementingthis,couldlooklikethefollowing:

Here,awebpageopensaWebSocket(ws://)toaHUB(amessageconcentrator)whichinturntalkstoadatabase.Oneachnewconnection,theHUBlogsinandonsuccessfulloginopensaWebSocketconnectiontothewebpage.Itthensendsallnewmessagesthathaveaccumulatedforthelogged-inusersincethelasttimetheyretrievedtheirmessages.Afterthat,theHUBwaitsfornewmessagesandpushesthemtothewebpageastheyarrive.

Thedatabaseparthastwotables,theuser_infotableandmessagetable.Thefollowingcodewillcreatetheuser_infotable:

CREATETABLEuser_info(

usernametextprimarykey,

pwdhashtextnotnull,—base64encodedmd5hashofpassword

emailtext,

friend_listtext[],—listofbuddiesusernames

friends_onlybooleannotnulldefaultfalse

);

Thefollowingcodewillcreatethemessagetable:

CREATETABLEmessage(

from_usertextnotnullreferencesuser_info(username),

sent_attimestampnotnulldefaultcurrent_timestamp,

to_usertextnotnullreferencesuser_info(username),

read_attimestamp,—whenwasthisretrievedbyto_user

msg_bodytextnotnull,

delivery_statustextnotnulldefault'outgoing'—('sent',"failed")

);

Asthisisstillanall-in-onedatabaseimplementation,thedatabasefunctionscorrespondingtoapplicationmethodsareverysimple.

Thefollowingcodewillcreateauser:

CREATEorREPLACEFUNCTIONnew_user(

INi_usernametext,INi_pwdhashtext,INi_emailtext,

OUTstatusint,OUTmessagetext)

AS$$

BEGIN

INSERTINTOuser_info(username,pwdhash,email)

VALUES(i_username,i_pwdhash,i_email);

status=200;

message='OK';

EXCEPTIONWHENunique_violationTHEN

status=500;

message='USEREXISTS';

END;


Thismethodjustfailswhentheuserisalreadydefined.Amorereal-lifefunctionwouldproposealistofavailableusernamesinthiscase.

Thefollowingmethodforloginreturnsstatus500forfailureand200or201forsuccess.Ifthemethodreturns201,itmeansthatthereareunreadmessagesforthisuser:

CREATEORREPLACEFUNCTIONlogin(

INi_usernametext,INi_pwdhashtext,


AS$$

BEGIN

PERFORM1FROMuser_info

WHERE(username,pwdhash)=(i_username,i_pwdhash);

IFNOTFOUNDTHEN

status=500;

message='NOTFOUND';

return;

ENDIF;

PERFORM1FROMmessage

WHEREto_user=i_username

ANDread_atISNULL;

IFFOUNDTHEN

status=201;

message='OK.NEWMESSAGES';

ELSE

status=200;

message='OK.NOMESSAGES';

ENDIF;

END;


Thefollowingaretheothertwousermethodsforchangingthefriends,listandtellingthesystemwhethertheywanttoreceivemailsthatareonlyfromfriends.Errorcheckingisomittedhereforbrevity:

CREATEorREPLACEFUNCTIONset_friends_list(

INi_usernametext,INi_friends_listtext[],


AS$$

BEGIN

UPDATEuser_info

SETfriend_list=i_friends_list

WHEREusername=i_username;

status=200;

message='OK';

END;


CREATEorREPLACEFUNCTIONmsg_from_friends_only(

INi_usernametext,INi_friends_onlyboolean,OUTstatusint,OUT

messagetext)

AS$$

BEGIN

UPDATEuser_infoSETfriends_only=i_friends_only

WHEREusername=i_username;

status=200;

message='OK';

END;


Thesend_message()functionisusedformessaging,whichsimplysendsmessages,asshowninthefollowingcode:

CREATEorREPLACEFUNCTIONsend_message(

INi_from_usertext,INi_to_usertext,INi_messagetext,


AS$$

BEGIN

PERFORM1FROMuser_info

WHEREusername=i_to_user

AND(NOTfriends_onlyORfriend_list@>ARRAY[i_from_user]);

IFNOTFOUNDTHEN

status=400;

message='SENDINGFAILED';

RETURN;

ENDIF;

INSERTINTOmessage(from_user,to_user,msg_body,delivery_status)

VALUES(i_from_user,i_to_user,i_message,'sent');

status=200;

message='OK';

EXCEPTION

WHENforeign_key_violationTHEN

status=500;

message='FAILED';

END;


Thefunctionusedforretrievingmessages,isasfollows:

CREATEorREPLACEFUNCTIONget_new_messages(

INi_usernametext,

OUTo_statusint,OUTo_message_texttext,

OUTo_from_usertext,OUTo_sent_attimestamp)

RETURNSSETOFRECORD

AS$$

BEGIN

FORo_status,o_message_text,o_from_user,o_sent_atIN

UPDATEmessage

SETread_at=CURRENT_TIMESTAMP,

delivery_status='read'

WHEREto_user=i_usernameANDread_atISNULL

RETURNING200,msg_body,from_user,sent_at

LOOP

RETURNNEXT;

ENDLOOP;

END;


Wearealmostdonewiththedatabasepartofoursimpleserver.Tofinishitup,weneedtodosomeinitialperformancetuning,andforthat,weneedsomedatainourtables.Theeasiestwayistousethegenerate_series()functiontogeneratealistofnumbers,whichwewilluseasusernames.Forourinitialtesting,nameslike7or42areasgoodasBob,Mary,orJill:

hannu=#SELECTnew_user(generate_series::text,'pwd',generate_series::text

||'@pg.org')

hannu-#FROMgenerate_series(1,100000);

hannu=#WITHns(n,len)AS(

hannu(#SELECT*,(random()*10)::intFROM

generate_series(1,100000))

hannu-#SELECTset_friends_list(ns.n::text,

hannu(#ARRAY((SELECT(random()*100000)::int

hannu(#FROMgenerate_series(1,len)))::text[]

hannu(#)

hannu-#FROMns;

Now,wehave100,000userswith0to10friendseach,foratotalof501,900friends.

hannu=#SELECTcount(*)FROM(SELECTusername,unnest(friend_list)FROM

user_info)a;

-[RECORD1]-

count|501900

Now,let’ssendeachofthefriendsamessage:

hannu=#SELECTsend_message(username,unnest(friend_list),'hellofriend!')

FROMuser_info;

Lookhowfastwecanretrievethemessages:

hannu=#SELECTget_new_messages('50000');

get_new_messages

----------------------------------------------------------

(200,"hellofriend!",49992,"2012-01-0902:23:28.470979")

(200,"hellofriend!",49994,"2012-01-0902:23:28.470979")

(200,"hellofriend!",49995,"2012-01-0902:23:28.470979")

(200,"hellofriend!",49996,"2012-01-0902:23:28.470979")

(200,"hellofriend!",49997,"2012-01-0902:23:28.470979")

(200,"hellofriend!",49999,"2012-01-0902:23:28.470979")

(200,"hellofriend!",50000,"2012-01-0902:23:28.470979")

(7rows)

Time:763.513ms

Spendingalmostasecondgettingsevenmessagesseemsslow,soweneedtooptimizeabit.

Thefirstthingtodoistoaddindexesforretrievingthemessages:

hannu=#CREATEINDEXmessage_from_user_ndxONmessage(from_user);

CREATEINDEX

Time:4341.890ms

hannu=#CREATEINDEXmessage_to_user_ndxONmessage(to_user);

CREATEINDEX

Time:4340.841ms

Andcheckifthishelpedtosolveourproblem:


get_new_messages

----------------------------------------------------------

(200,"hellofriend!",51993,"2012-01-0902:23:28.470979")

(200,"hellofriend!",51994,"2012-01-0902:23:28.470979")

(200,"hellofriend!",51996,"2012-01-0902:23:28.470979")

(200,"hellofriend!",51997,"2012-01-0902:23:28.470979")

(200,"hellofriend!",51998,"2012-01-0902:23:28.470979")

(200,"hellofriend!",51999,"2012-01-0902:23:28.470979")

(200,"hellofriend!",52000,"2012-01-0902:23:28.470979")

(7rows)

Time:2.949ms

Muchbetter—indexedlookupsare300timesfasterthansequentialscans,andthisdifferencewillgrowastablesgetbigger!

Asweareupdatingthemessagesandsettingtheirstatustoread,itisalsoagoodideatosetfillfactortosomethinglessthan100percentandthisisshowninthefollowingexample:

NoteFillfactortellsPostgreSQLnottofillupdatabasepagescompletely,buttoleavesomespaceforHOTupdates.WhenPostgreSQLupdatesarow,itonlymarkstheoldrowfordeletionandaddsanewrowtothedatafile.Iftherowthatisupdatedonlychangesunindexedfields,andthereisenoughroominthepagetostoreasecondcopy,aHOTupdatewillbedoneinstead.Inthiscase,thecopycanbefoundusingoriginalindexpointerstothefirstcopy,andnoexpensiveindexupdatesaredonewhileupdating.Youcanreadmoreaboutthefillfactorathttp://www.postgresql.org/docs/current/static/sql-createtable.html

hannu=#ALTERTABLEmessageSET(fillfactor=90);

ALTERTABLE

Time:75.729ms

http://www.postgresql.org/docs/current/static/sql-createtable.html

hannu=#CLUSTERmessage_from_user_ndxONmessage;

CLUSTER

Time:9797.639ms


get_new_messages

----------------------------------------------------------

(200,"hellofriend!",55014,"2012-01-0902:23:28.470979")

(200,"hellofriend!",55016,"2012-01-0902:23:28.470979")

(200,"hellofriend!",55017,"2012-01-0902:23:28.470979")

(200,"hellofriend!",55019,"2012-01-0902:23:28.470979")

(200,"hellofriend!",55020,"2012-01-0902:23:28.470979")

(200,"hellofriend!",55021,"2012-01-0902:23:28.470979")

(200,"hellofriend!",55022,"2012-01-0902:23:28.470979")

(7rows)

Time:1.895ms

Stillbetter!Thefillfactormadetheget_new_messages()functionanother20to30percentfaster,thankstoenablingthefasterHOTupdates!

Dealingwithsuccess–splittingtablesovermultipledatabasesNow,let’srollforwardalittleintimeandassumeyouhavebeensuccessfulenoughtoattracttensofthousandsofusersandyoursingledatabasestartscreakingundertheload.

Mygeneralruleofthumb,istostartplanningforabiggermachine,orsplittingthedatabase,whenyouareover80percentutilizationatleastforafewhoursaday.It’sgoodtohaveaplanearlier,butnowyouhavetostartdoingsomethingaboutreallycarryingouttheplan.

Whatexpansionplansworkandwhen?Thereareacoupleofpopularwaystogrowdatabase-backedsystems.Dependingonyourusecase,notallwayswillwork.

MovingtoabiggerserverIfyoususpectthatyouarenearyourtoploadfortheserviceorproduct,youcansimplymovetoamorepowerfulserver.Thismaynotbethebestlong-timescalingsolutionifyouarestillinthemiddle,oreveninthebeginningofyourgrowth.Youwillrunoutofbiggermachinestobuylongbeforeyouaredone.Serversalsobecomedisproportionatelymoreexpensiveasthesizeincreases,andyouwillbeleftwithatleastonedifferent,andthusnoteasilyreplaceable,serveronceyouimplementaproperscalingsolution.

Ontheotherhand,thiswillworkforsometimeandisoftentheeasiestwaytogetsomeheadroomwhileimplementingrealscalingsolutions.

Master-slavereplication–movingreadstoslaveMaster-slavereplication,eithertrigger-basedorWAL-based,worksreasonablywellincaseswherethelargemajorityofthedatabaseaccessesarereads.Somethingsthatfallunderthiscase,arewebsitecontentmanagers,blogs,andotherpublishingsystems.

Asourchatsystemhasmoreorlessa1:1ratioofwritesandreads,movingreadstoaseparateserverwillbuyusnothing.Thereplicationitselfismoreexpensivethanthepossiblewinfromreadingfromasecondserver.

MultimasterreplicationMultimasterreplicationisevenworsethanmaster-slave(s)whentheproblemisscalingawrite-heavyworkload.Ithasalltheproblemsofmaster-slave,plusitintroducesextraloadviacross-partitionlockingorconflictresolutionrequirements,whichfurtherslowsdownthewholecluster.

DatapartitioningacrossmultipleserversTheobvioussolutiontoscalingwritesistosplitthembetweenseveralservers.Ideallyyoucouldhave,forexample,fourserversandeachofthemgettingexactlyonefourthoftheload.

Inthiscase,eachserverwouldholdaquarterofusersandmessages,andserveaquarterofallrequests.

Tomakethechangetransparentfordatabaseclients,weintroducealayerofproxydatabases.Theseproxydatabasescaneitherresideonthesamehostsasthepartitiondatabasesorbeontheirownhost.Theroleoftheproxydatabasesistopretendtobethedatabaseforclients,butinfactdelegatetherealworktopartitionsbycallingtherightfunctionintherightpartitiondatabase.

Thisclienttransparencyisnotterriblyimportantifyouhavejustoneapplicationaccessingthedatabase.Ifyoudid,youcouldthendothesplittingintheclientapplication.Itbecomesveryhandyasyoursystemgrowstohaveseveralapplications,perhapsusingmanydifferentplatformsandframeworksontheclientside.

Havingaseparatelayerofproxydatabasesenableseasymanagementofdatasplitting,sothattheclientapplicationsdon’tneedtoknowanythingabouttheunderlyingdataarchitecture.Theyjustcallthefunctionstheyneedandthat’salltheyneedtoknow.Infact,youcanswitchoutthewholedatabasestructurewithouttheclientsevernoticinganything,exceptthebetterperformancefromthenewarchitecture.

Moreonhowexactlytheproxyworkslater.Fornow,letustacklesplittingthedata.

SplittingthedataIfwesplitthedata,weneedasimpleandefficientwaytodeterminewhichserverstoreseachdatarow.Ifthedatahadanintegerprimarykey,youcouldjustgoround-robin,storethefirstrowonthefirstserver,thesecondrowonthesecond,andsoon.Thiswouldgiveyouafairlyevendistribution,evenwhenrowswithcertainIDsaremissing.

Thepartitioningfunctionforselectingbetweenfourserverswouldbesimply:

partition_nr=id&3

Thepartitioningmask3(binary11)isforthefirsttwobits.Foreightpartitions,youwoulduse7(binary111),andfor64serversitwouldbe63(00111111).Itisnotaseasywiththingslikeusernames,whereputtingallnamesstartingwithanAfirst,Bsecond,andsoon,doesnotproduceanevendistribution.

Turningtheusernameintoafairlyevenlydistributedintegerviathehashfunctionsolvesthisproblemandcanbeuseddirectlytoselectthepartition.

partition_nr=hashtext(username)&3

Thiswoulddistributetheusersinthefollowingmanner:

hannu=#SELECTusername,hashtext(username)&3aspartition_nrFROM

user_info;

-[RECORD1]+--------

username|bob

partition_nr|1

-[RECORD2]+--------

username|jane

partition_nr|2

-[RECORD3]+--------

username|tom

partition_nr|1

-[RECORD4]+--------

username|mary

partition_nr|3

-[RECORD5]+--------

username|jill

partition_nr|2

-[RECORD6]+--------

username|abigail

partition_nr|3

-[RECORD7]+--------

username|ted

partition_nr|3

-[RECORD8]+--------

username|alfonso

partition_nr|0

So,partition0getsuseralfonso,partition1bobandtom,partition2janeandjill,andpartition3getsmary,abigail,andted.Thedistributionisnotexactlyonefourthtoeachpartition;butasthenumberofpartitionsincrease,itwillbeprettyclosewherethisactuallymatters.

IfwehadnoPL/Proxylanguage,wecouldwritethepartitioningfunctionsinthemostuntrustedPLlanguages.Forexample,asimpleloginproxyfunctionwritteninPL/Pythonulookslikethis:

CREATEORREPLACEFUNCTIONpylogin(



AS$$

importpsycopg2

partitions=[

'dbname=chap10p0port=5433',




]

partition_nr=hash(i_username)&3

con=psycopg2.connect(partitions[partition_nr])

cur=con.cursor()

cur.execute('select*fromlogin(%s,%s)',(i_username,i_pwdhash))

status,message=cur.fetchone()

return(status,message)

$$LANGUAGEplpythonuSECURITYDEFINER;

Here,wedefinedasetoffourpartitiondatabases,givenbytheirconnectstringsandstored

asalistinvariablepartitions.

Whenexecutingthefunction,wefirstevaluatethehashfunctionontheusernameargument(hash(i_username))andextracttwobitsfromit(&3)togetanindexintothepartitions’list(thepartitionnumber)forexecutingeachcall.

Then,weopenaconnectiontoapartitiondatabaseusingtheconnectstringselectedbythepartitionnumber(con=psycopg2.connect(partitions[partition_nr])).

Finally,weexecutearemotequeryinthepartitiondatabaseandreturntheresultsofthistothecallerofthisproxyfunction.

Thisworksreasonablywell,ifimplementedlikethis,butitalsohasatleasttwoplaceswhereitissuboptimal:

First,itopensanewdatabaseconnectioneachtimethefunctioniscalled,whichkillsperformanceSecond,itisamaintenancenightmareifyouhard-wirethepartitioninformationinfull,inallfunctions

Theperformanceproblemcanbesolvedbycachingtheopenconnections,andthemaintenanceproblemcanbesolvedbyhavingasinglefunctionreturningthepartitioninformation.However,evenwhenwedothesechangesandstaywithPL/Pythonuforpartitioning,wewillstillbedoingalotofcopyandpasteprogrammingineachofourproxyfunctions.

Oncewehadreachedtheprecedingconclusions,whengrowingourdatabasesystemsatSkype,thenextlogicalstepwasquiteobvious.Weneededaspecialpartitioninglanguage,whichwoulddojustthisonething—callingremoteSQLfunctions,andthenmakeitasfastaspossible.Andthus,thePL/Proxydatabasepartitioninglanguagewasborn.

PL/Proxy–thepartitioninglanguageTherestofthischapterisdevotedtothePL/Proxylanguage.First,wewillinstallit.Then,wewilllookatitssyntaxandwaystoconfigurethepartitionsforitsuse.Finally,wewilldiscusshowtodotheactualdatamigrationfromasingledatabasetoapartitionedoneandthenlookatseveralusageexamples.

InstallingPL/ProxyIfyouareonDebian,Ubuntu,oraRedHatvariant,installingthelanguageiseasy.

First,youhavetoinstalltherequiredpackagesonyouroperatingsystem:

sudoapt-getinstallpostgresql-9.4-plproxy

Or:

sudoyuminstallplproxy94

IfyouneedtoinstallPL/Proxyfromthesource,youcandownloaditfromhttp://pgfoundry.org/projects/plprox,extractthesourcesinthecontribfolderofyourPostgreSQLsourcetreeandrunmakeandmakeinstall.

ToinstallPL/Proxyyoushouldruntheplproxy.sqlfile,whichispartofthesourcecodeorthepackageyouinstalled.

ThePL/ProxylanguagesyntaxThePL/Proxylanguageitselfisverysimple.ThepurposeofaPL/Proxyfunctionistohandofftheprocessingtoanotherserver,sothatthelanguageonlyneedssixstatements:

CONNECTorCLUSTERandRUNONforselectingthetargetdatabasepartitionSELECTandTARGETforspecifyingthequerytorunSPLITforsplittinganARRAYargumentbetweenseveralsubarraysforrunningonmultiplepartitions

CONNECT,CLUSTER,andRUNONThefirstgroupofstatementshandlestheremoteconnectivitytothepartitions.Thehelpdetermineswhichdatabasetorunthequeryon.YouspecifytheexactpartitiontorunthequeryusingCONNECT:

CONNECT'connectstring';

Here,connectstringdeterminesthedatabasetorun.connectstringisthestandardPostgreSQLconnectstringyouwouldusetoconnecttothedatabasefromaclientapplication,forexample:dbname=p0port=5433username=testhost=localhost.

Or,youcanspecifyanameusingCLUSTER:

CLUSTER'usercluster';

Finally,youcanspecifyapartitionnumberusingRUNON:

RUNONpart_func(arg[,...]);

http://pgfoundry.org/projects/plprox

part_func()canbeanyexistingoruser-definedPostgreSQLfunctionreturninganinteger.PL/ProxycallsthatfunctionwiththegivenargumentsandthenusesNlowerbitsfromtheresulttoselectaconnectiontoaclusterpartition.

TherearetwomoreversionsoftheRUNONstatement:

RUNONANY;

Thismeansthatthefunctioncanbeexecutedonanypartitioninacluster.Thiscanbeusedwhenalltherequireddataforafunctionispresentonallpartitions.

Theotherversionis:

RUNONALL;

Thisrunsthestatementonallpartitionsinparallelandthenreturnsaconcatenationofresultsfromthepartitions.Thishasatleastthreemainuses:

Forcaseswhenyoudon’tknowwheretherequireddatarowis,likewhengettingdatausingnon-partitionkeys.Forexample,gettingauserbyitse-mailwhenthetableispartitionedbyusername.Runningaggregatefunctionsoverlargersubsetsofdata,saycountingallusers.Forexample,gettingalltheuserswhohaveacertainuserintheirfriends’lists.Manipulatingdatathatneedstobethesameonallpartitions.Forexample,whenyouhaveapricelistthatotherfunctionsareusing,thenonesimplewaytomanagethispricelistisusingaRUNONALLfunction.

SELECTandTARGETThedefaultbehaviorofaPL/Proxyfunction,ifnoSELECTorTARGETispresent,istocallthefunctionwiththeexactsamesignatureasitselfintheremotepartition.

Supposewehavethefunction:

CREATEORREPLACEFUNCTIONlogin(



AS$$

CONNECT'dbname=chap10host=10.10.10.1';

$$LANGUAGEplproxySECURITYDEFINER;

Ifitisdefinedinschemapublic,thefollowingcallselect*fromlogin('bob','secret')connectstothedatabasechap10onhost10.10.10.1andrunsthefollowingSQLstatementthere:

SELECT*FROMpublic.login('bob','secret')

Thisretrievestheresultandreturnsittoitscaller.

Ifyoudon’twanttodefineafunctioninsidetheremotedatabase,youcansubstitutethedefaultselect*from<thisfunction>(<arg1>,...)callwithyourownbywritingitinthefunctionbodyofthePL/Proxyfunction:

CREATEORREPLACEFUNCTIONget_user_email(i_usernametext)

RETURNSSETOFtextAS$$

CONNECT'dbname=chap10host=10.10.10.1';

SELECTemailFROMuser_infowhereusername=i_username;


OnlyasingleSELECTissupported;foranyother,ormorecomplexSQLstatements,youhavetowritearemotefunctionandcallit.

Thethirdoption,istostillcallafunctionsimilartoitself,butnameddifferently.Forexample,ifyouhaveaproxyfunctiondefinednotinaseparateproxydatabase,butinapartition,youmaywantittotargetthelocaldatabaseforsomedata:

CREATEORREPLACEFUNCTIONpublic.get_user_email(i_usernametext)RETURNS

SETOFtextAS$$

CLUSTER'messaging';

RUNONhashtext(i_username);

TARGETlocal.get_user_email;


Inthissetup,thelocalversionofget_user_email()isinschemalocalonallpartitions.Therefore,ifoneofthepartitionsconnectsbacktothesamedatabasethatitisdefinedin,itavoidscircularcalling.

SPLIT–distributingarrayelementsoverseveralpartitionsThelastPL/Proxystatementisforcaseswhereyouwantsomebiggerchunkofworktobedoneinappropriatepartitions.

Forexample,ifyouhaveafunctiontocreateseveralusersinonecallandyoustillwanttobeabletouseitafterpartitioning,theSPLITstatementisawaytotellPL/Proxytosplitthearraysbetweenthepartitionsbasedonthepartitioningfunction:

CREATEorREPLACEFUNCTIONcreate_new_users(

INi_usernametext[],INi_pwdhashtext[],INi_emailtext[],

OUTstatusint,OUTmessagetext)RETURNSSETOFRECORD

AS$$

BEGIN

FORiIN1..array_length(i_username,1)LOOP

SELECT*

INTOstatus,message

FROMnew_user(i_username[i],i_pwdhash[i],i_email[i]);

RETURNNEXT;

ENDLOOP;

END;


ThefollowingPL/Proxyfunctiondefinition,createdontheproxydatabase,canbeusedtosplitthecallsacrossthepartitions:

CREATEorREPLACEFUNCTIONcreate_new_users(

INi_usernametext[],INi_pwdhashtext[],INi_emailtext[],

OUTstatusint,OUTmessagetext)RETURNSSETOFRECORD

AS$$

CLUSTER'messaging';

RUNONhashtext(i_username);

SPLITi_username,i_pwdhash,i_email;


Itwouldbecalledbysendinginthreearraystothefunction:

SELECT*FROMcreate_new_users(

ARRAY['bob','jane','tom'],

ARRAY[md5('bobs_pwd'),md5('janes_pwd'),md5('toms_pwd')],

ARRAY['[email protected]','[email protected]','[email protected]']

);

Itwillresultintwoparallelcallstopartitions1and2(asusinghashtext(i_username)bobandtommaptopartition1andmarytopartition2ofthetotalforthepartitions,asexplainedearlier),withthefollowingargumentsforpartition1:


ARRAY['bob','tom'],

ARRAY['6c6e5b564fb0b192f66b2a0a60c751bb','edcc36c33f7529f430a1bc6eb7191dfe'

],

ARRAY['[email protected]','[email protected]']

);

Andthisforpartition2:


ARRAY['jane'],

ARRAY['cbbf391d3ef4c60afd851d851bda2dc8'],

ARRAY['[email protected]']

);

Then,itreturnsaconcatenationoftheresults:

status|message

--------+---------

200|OK

200|OK

200|OK

(3rows)

ThedistributionofdataFirst,whatisaclusterinPL/Proxy?Well,theclusterisasetofpartitionsthatmakeupthewholedatabase.Eachclusterconsistsofanumberofpartitions,asdeterminedbytheclusterconfiguration.Eachpartitionisuniquelyspecifiedbyitsconnectstring.Thelistofconnectionstringsiswhatmakesupacluster.Thepositionofthepartitioninthislistiswhatdeterminesthepartitionnumber,sothefirstelementinthelistispartition0,thesecondpartitionis1,andsoon.

ThepartitionisselectedbytheoutputoftheRUNONfunction,andthenmaskedbytherightnumberofbitstomapitonthepartitions.So,ifhashtext(i_username)returns14andtherearefourpartitions(2bits,maskbinary11or3indecimal),thepartitionnumberwillbe14and3=2,andthefunctionwillbecalledonpartition2(startingfromzero),whichisthethirdelementinthepartitionlist.

NoteTheconstraintthatthenumberofpartitionshastobeapoweroftwomayseemanunnecessaryrestrictionatfirst,butitwasdoneinordertomakesurethatitis,andwillremaintobe,easytoexpandthenumberofpartitionswithouttheneedtoredistributeallthedata.

Forexample,ifyoutriedtomovefromthreepartitionstofour,mostlikelythreefourthsofthedatarowsinpartitions0to2havetobemovedtonewpartitionstoevenlycover0to3.Ontheotherhand,whenmovingfromfourtoeightpartitions,thedataforpartitions0and1isexactlythesamethatwaspreviouslyonpartition0,2-3istheold1andsoon.Thatis,yourdatadoesnotneedtobemovedimmediately,andhalfofthedatadoesnotneedtobemovedatall.

Theactualconfigurationofthecluster,thedefinitionofpartitions,canbedoneintwoways,eitherbyusingasetoffunctionsinschemaplproxy,oryoucantakeadvantageoftheSQL/MEDconnectionmanagement(SQL/MEDisavailablestartingatPostgreSQL8.4andabove).YoucanreadmoreaboutSQL/MEDherehttps://wiki.postgresql.org/wiki/SQL/MED

ConfiguringthePL/ProxyclusterusingfunctionsThisistheoriginalwaytoconfigurePL/Proxy,whichworksonallversionsofPostgreSQL.Whenaqueryneedstobeforwardedtoaremotedatabase,thefunctionplproxy.get_cluster_partitions(cluster)isinvokedbyPL/Proxytogettheconnectionstringtouseforeachpartition.

Thefollowingfunctionisanexample,whichreturnsinformationforaclusterwithfourpartitions,p0top3:

CREATEORREPLACEFUNCTIONplproxy.get_cluster_partitions(cluster_name

text)

RETURNSSETOFtextAS$$

BEGIN

IFcluster_name='messaging'THEN

RETURNNEXT'dbname=p0';




ELSE

RAISEEXCEPTION'Unknowncluster';

ENDIF;

END;

$$LANGUAGEplpgsql;

Aproductionapplicationmightquerysomeconfigurationtables,orevenreadsomeconfigurationfilestoreturntheconnectionstrings.Onceagain,thenumberofpartitionsreturnedmustbeapoweroftwo.Ifyouareabsolutelysurethatsomepartitionsareneverused,youcanreturnemptystringsforthese.

Wealsoneedtodefineaplproxy.get_cluster_version(cluster_name)function.Thisiscalledoneachrequestandiftheclusterversionhasnotchanged,theoutputfroma

https://wiki.postgresql.org/wiki/SQL/MED

cachedresultfromplproxy.get_cluster_partitionscanbereused.So,itisbesttomakesurethatthisfunctionisasfastaspossible:

CREATEORREPLACEFUNCTIONplproxy.get_cluster_version(cluster_nametext)

RETURNSint4AS$$

BEGIN

IFcluster_name='messaging'THEN

RETURN1;

ELSE

RAISEEXCEPTION'Unknowncluster';

ENDIF;

END;

$$LANGUAGEplpgsql;

Thelastfunctionneededisplproxy.get_cluster_config,whichenablesyoutoconfigurethebehaviorofPL/Proxy.Thissamplewillsettheconnectionlifetimeto10minutes:

CREATEORREPLACEFUNCTIONplproxy.get_cluster_config(

incluster_nametext,

outkeytext,

outvaltext)

RETURNSSETOFrecordAS$$

BEGIN

—letsusesameconfigforallclusters

key:='connection_lifetime';

val:=10*60;

RETURNNEXT;

RETURN;

END;

$$LANGUAGEplpgsql;

ConfiguringthePL/ProxyclusterusingSQL/MEDSinceversion8.4,PostgreSQLhassupportforanSQLstandardformanagementofexternaldata,usuallyreferredtoasSQL/MED.SQL/MEDissimplyastandardwaytoaccessadatabase.Usingfunctionstoconfigurepartitionsisarguablyinsecure,asanycallerofplproxy.get_cluster_partitions()canlearnconnectionstringsforpartitionsthatmaycontainsensitiveinfolikepasswords.PL/ProxyalsoprovidesawaytodotheclusterconfigurationusingSQL/MED,whichfollowsthestandardSQLsecuritypractices.

Thesameconfiguration,asdiscussedearlier,whendoneusingSQL/MED,isasfollows:

1. First,createaforeigndatawrappercalledplproxy:

proxy1=#CREATEFOREIGNDATAWRAPPERplproxy;

2. Then,createanexternalserverthatdefinesboththeconnectionoptionsandthepartitions:

proxy1=#CREATESERVERmessagingFOREIGNDATAWRAPPERplproxy

proxy1-#OPTIONS(connection_lifetime'1800',

proxy1(#p0'dbname=p0',



proxy1(#p3'dbname=p3'

proxy1(#);

CREATESERVER

3. Then,grantusageonthisservertoeitherPUBLIC,soalluserscanuseit:

proxy1=#CREATEUSERMAPPINGFORPUBLICSERVERmessaging;

CREATEUSERMAPPING

Or,tosomespecificusersorgroups:

proxy1=#CREATEUSERMAPPINGFORbobSERVERmessaging

proxy1-#OPTIONS(user'plproxy',password'very.secret');

CREATEUSERMAPPING

4. Finally,grantusageontheclustertotheuserswhoneedtouseit:

proxy1=#GRANTUSAGEONFOREIGNSERVERmessagingTObob;

GRANT

NoteMoreinfoonSQL/MED,asimplementedinPostgreSQL,canbefoundathttp://www.postgresql.org/docs/current/static/sql-createforeigndatawrapper.html.

http://www.postgresql.org/docs/current/static/sql-createforeigndatawrapper.html

MovingdatafromthesingletothepartitioneddatabaseIfyoucanschedulesomedowntimeandyournewpartitiondatabasesareasbigasyouroriginalsingledatabase,theeasiestwaytopartitionthedataistomakeafullcopyofeachofthenodesandthensimplydeletetherowsthatdonotbelongtothepartition:

pg_dumpchap10|psqlp0

psqlp0-c'deletefrommessagewherehashtext(to_user)&3<>0'

psqlp0-c'deletefromuser_infowherehashtext(username)&3<>0'

Repeatthisforpartitionsp1top3,eachtimedeletingrowswhichdon’tmatchthepartitionnumber(psqlchap10p1-c'delete…&3<>1).

NoteRemembertovacuumwhenyouarefinisheddeletingtherows.PostgreSQLwillleavethedeadrowsinthedatatables,sodoalittlemaintenancewhileyouhavesomedowntime.

Whentryingtodeletefromuser_info,youwillnoticethatyoucan’tdoitwithoutdroppingaforeignkeyfrommessage.from_user.

Here,wecoulddecidethatitisOkaytokeepthemessagesonthereceiverspartitiononly,andifneeded,thatthesentmessagescanberetrievedusingaRUNONALLfunction.So,wewilldroptheforeignkeyfrommessages.from_user.

psqlp0-c'altertablemessagedropconstraintmessage_from_user_fkey'

Thereareotheroptions,whensplittingthedata,thatrequirelessdiskspaceusageforadatabasesystem,ifyouarewillingtodomoremanualwork.

Forexample,youcancopyoverjusttheschemausingpg_dump-sandthenuseCOPYfromanSQLstatementtomoveoverjusttheneededrows:

pg_dump-schap10|psqlp0

psqlchap10-c"COPY(select*frommessagewherehashtext(to_user)&3=

0)TOstdout"|psqlp0-c'COPYmessagesFROMstdin'

OrevensetupaspeciallydesignedLondistereplicaanddotheswitchfromasingledatabasetoapartitionedclusterinonlyseconds,oncethereplicahasreachedastablestate.

ConnectionPoolingPL/Proxydoesnotincludeaconnectionpooleranditisagoodideatouseonelikepgbouncerandpgpool.Aconnectionpoolerisautilitythathelpsyoureducetheoperatingcostofdatabase,whenitslargenumberofphysicalconnectionsarepullingperformancedown.

PL/Proxyopensaconnectiontoeachpartitionfromeachbackendprocessandalargenumberofconnectionscanbringtheperformanceoftheserverdown.Usingtheconnectionpoolwillallowyoutomultiplexalotofclientconnectionsoverasmallnumberofdatabaseconnections.Thepgbouncerisarecommendedconnectionpoolerbecauseit’slightweightandveryeasytosetup.PL/ProxyattemptstoconnecttopgbouncerusingthesamedatabaseconnectioninterfacethatitusestoconnecttoanyPostgreSQLdatabase.TheclientapplicationsuppliestheIPaddressofthehostrunningpgbouncerandtheportnumberonwhichpgbouncerislisteningforconnections.

SummaryInthischapter,wehavegoneovertheprocessofdatabaseshardingfordatabasesthataretoobigtotakethewriteloadonasinglehost,orwhereyoujustwanttohavetheaddedresilienceofhavingasystemwhereonehostbeingdowndoesnotbringthewholesystemdown.

Inshort,theprocessis:

DecidewhichtablesyouwanttosplitovermultiplehostsDefineapartitioningkeyAddthepartitiondatabasesandmovethedataSetuptheproxyfunctionsforallthefunctionsaccessingthosetablesWatchforalittlewhilethateverythingisworkingRelax

WealsotookabrieflookatusingPL/ProxyforsimpleremotequeriestootherPostgreSQLdatabases,whichmaybehandyforsometasks,evenafterthenewForeignDataWrapper(FDW)functionalityinPostgreSQLreplaceditformanyuses.

WhilePL/Proxyisnotforeveryone,itmaywellsavethedayifyouaresuddenlyfacedwithrapiddatabasegrowthandhavetheneedforaneasyandcleanwaytospreadthedatabaseovermanyhosts.

Inthenextchapter,wewilllookatthePL/Perlprogramminglanguage.

Chapter11.PL/Perl–PerlProceduralLanguagePerlisafeature-richlanguage,whichhasbeenaroundforalongtime.PostgreSQLallowsyoutowritePerlroutinesthatarestoredandexecutedinsidethedatabase.ThisabilityisquiteuniquetoPostgreSQLandallowsyoutodoalotofcoolthings,suchasusingPerl’stextmanipulationfeaturesinsideadatabase.PL/PerlisoneofthemanylanguagesthatPostgreSQLsupportsforwritingserver-sideroutines.

Asdiscussedintheearlierchapters,PostgreSQLsupportstrustedanduntrustedlanguages.PL/Perlisavailableinboththeseflavors.Thetrustedversionrunsinsideasafecontainerand,therefore,nottheentiresetoffamiliarnativePerloperationsisallowed.


WhentousePL/PerlHowtoinstallandwriteabasicfunctionPassingargumentstoandfromPL/PerlfunctionsWritingtriggersinPL/PerlAbriefintroductiontountrustedPL/Perl

WhentousePL/PerlWewillbrieflydiscusshowtocreatePerlfunctionsandusethemintriggers.SomeofyoumightaskwhenitisbeneficialtousePL/Perl,sincePostgreSQLsupportsavarietyoflanguagesthatcanbeusedtocreatetriggersorwritestoredprocedures/functions.

YoumightbemorefamiliarwithalanguagesuchasPerlthanwithPythonorTcl.Thisisaprettygoodreasontochooseacertainlanguageovertheother.

However,iftheperformanceofthefunctionisoneofyourconsiderations,thenyoumightwanttochoosealanguagebasedonthenatureofthecodeyouwanttowrite.

Ingeneral,PL/PerlwilloutperformPL/pgSQL,ifthefocusofthestoredprocedureiscomputationaltasks,athematic,andstringparsingandmanipulation.However,PL/pgSQLwilloftenbeaclearwinnerifyouneedtoaccessthedatabase.PL/pgSQLiscloselytiedintothePostgreSQLexecutionengineandhasaverylowoverheadofrunningaquery,comparedtootherprocedurallanguages.

InstallingPL/PerlPL/PerlisnotinstalledbydefaultifyouusedthestandardsourcedistributiontoinstallPostgreSQL.IfyoucompilePostgreSQLfromthesource,youneedtoconfigurethescriptwiththe--with-perloption.

Ifyouusedabinarydistributiononyourplatform,youcannormallyinstallPL/Perlusingyourpackagemanager.Youcansearchforpostgresql-plperl,orasimilarpackagename,asitdiffersacrossthedistributions.OncePostgreSQLiscompiledwiththecorrectoption,oryouhaveinstalledtheappropriatepackage,youcancreatethePLlanguageusingthecreatelangutilityortheCREATELANGUAGEcommand,asshownhere:

$createlangplperltemplate1

Ortheuntrustedversion,suchasthefollowingcommand:

$createlangplperlutemplate1

hon

AsimplePL/PerlfunctionNow,let’swriteourfirstsimplePerlfunctiontomakesurethatPL/Perlisinstalledcorrectly.WewilluseasamplefunctionfromthePerlFAQs,athttp://perldoc.perl.org/perlfaq5.html#How-can-I-output-my-numbers-with-commas-added%3f,towriteaPL/Perlfunctionthataddscommastoanumber:

CREATEORREPLACEFUNCTIONcommafy(integer)RETURNStext

AS$$

local$_=shift;

1whiles/^([-+]?\d+)(\d{3})/$1,$2/;

return$_;

$$LANGUAGEplperl;

Thisfunctionusesasmartlywrittenregextoaddcommastoyournumbers.Let’stryandrunit:

testdb=#SELECTcommafy(1000000);

commafy

-----------

1,000,000

(1row)

ThecodeworksandthefunctionlookssimilartootherfunctionswehavebeenwritinginPL/pgSQLandPL/Python.TheCREATEFUNCTIONstatementcreatesafunction.Itneedsaname,functionargumenttypelist(youhavetouseparentheses,eveniftherearenoarguments),aresulttype,andalanguage.

ThefunctionbodyisjustananonymousPerlsubroutine.PostgreSQLpassesthisontoaPerlinterpreter,inordertorunthissubroutineandreturntheresults.Eachfunctioniscompiledoncepersession.PL/Perlfunctionargumentsarestoredin@_,justasinnormalPerlsubroutinesandyourcodecanhandlethemthesameway.

TheargumentspassedtothePL/PerlfunctionareconvertedtoUTF-8fromthedatabaseencoding,andthereturnvalueisconvertedfromUTF-8backtothedatabaseencoding.

TipUsingUTF-8fordatabaseencodingisencouragedaswell.Thiswillavoidtheoverheadofconvertingbackandforthbetweenencodings.

PL/Perlfunctionsruninascalarcontextandcannotreturnnon-scalartypessuchaslists.Youcanreturnthenon-scalartypes,suchasarrays,records,andsets,byreturningareference.

http://perldoc.perl.org/perlfaq5.html#How-can-I-output-my-numbers-with-commas-added%3f

Passingandreturningnon-scalartypesIfyoupassarraytypesasargumentstothePL/Perlfunction,theyarepassedastheblessedostgreSQL::InServer::ARRAYobjects.InPerl,blessassociatesanobjectwithaclass.Thisobjectcanbetreatedasanarrayreferenceorasastring.Ifyouhavetoreturnanarraytype,youmustreturnanarraybyreference.Let’stakealookatthefollowingexample:

CREATEORREPLACEFUNCTIONreverse(int[])RETURNSint[]

AS$$

my$arg=shift;#getthereferenceoftheargument

my@rev=reverse@{$arg};#reversethearray

return\@rev#returnthearrayreference

$$LANGUAGEplperl;

testdb=#selectreverse(ARRAY[1,2,3,4]);

reverse

-----------

{4,3,2,1}

(1row)

TheprecedingfunctionreversesanintegerarraypassedtothefunctionusingthereversefunctionofPerl.Youcantakealookatthecommentsinthecodetounderstandit.First,wegetthereferenceofthepassedargument.Wethenreversethearrayusingareferencenotation,@{$arg},[email protected],wereturnthereferenceofthearrayusingthebackslash.Ifyoutrytoreturnthearraydirectly,youwillgetanerror.

Let’stakealookatanotherfunctionthatconcatenatestwoarraysandthenreversestheirorder:

CREATEORREPLACEFUNCTIONconcat_reverse_arrays(int[],int[])RETURNS

int[]

AS$$

my$arr1=$_[0];

my$arr2=$_[1];

push(@{$arr1},@{$arr2});

my@reverse=reverse@{$arr1};

return\@reverse;

$$LANGUAGEplperl;

testdb=#selectconcat_reverse_arrays(ARRAY[1,2,3],ARRAY[4,5,6]);

concat_reverse_arrays

-----------------------

{6,5,4,3,2,1}

Again,thecodeisquitesimple.Ittakestwointegerarraysasparameters,concatenatesthemintoarr1,andreversestheorderusingstandardPerlfunctions.

PL/Perlcantakeargumentsofcompositetypesandcanalsoreturncompositetypes.Thecompositetypesarepassedasareferencetohashes,andthekeysofthehasharesimply

attributenamesofthepassedcomplextype.Let’stakealookatanexample:

CREATETABLEitem(item_namevarchar,item_priceint,discountint);

INSERTINTOitem(item_name,item_price)VALUES('macbook',1200);

INSERTINTOitem(item_name,item_price)VALUES('pen',5);

INSERTINTOitem(item_name,item_price)VALUES('fridge',1000);

CREATEORREPLACEFUNCTIONadd_discount(item)RETURNSitem

AS$$

my$item=shift;

if($item->{item_price}>=1000){

$item->{discount}=10;

}else{


}

return$item;

$$LANGUAGEplperl;

testdb=#selectadd_discount(item.*)fromitem;

add_discount

-------------------

(macbook,1200,10)

(pen,5,5)

(fridge,1000,10)

(3rows)

Theprecedingfunctionisprobablynotthebestusecaseforacomplextypeexample,butitdemonstratestheconceptssufficiently.First,wecreateatableandfillitupwithsomedata,exceptthediscountfield.Wethencreateafunctionthatispassedanitemtype,anditfillsupthediscountfieldbasedontheprice.Youcanseethatsincethecomplextypesarepassedashashreferences,weneedtousethe$item->{discount}notationtoaccessthekeys,whereeachkeycorrespondstoafieldinthetype.

YoucanalsoreturntheSETOFprimitiveandcomplextypesfromaPL/Perlfunction.Inthenextexample,wewilltrytoreturnaSETOFitemtypeandalsodemonstratehowtorunanSQLqueryinaPL/Perlfunction:

CREATEORREPLACEFUNCTIONset_dicounts()RETURNSSETOFitem

AS$$

my$rv=spi_exec_query('select*fromitem;');

my$nrows=$rv->{processed};

foreachmy$rn(0..$nrows-1){

my$item=$rv->{rows}[$rn];

if($item->{item_price}>=1000){


}else{


}

return_next($item);

}

returnundef;

$$LANGUAGEplperl;

postgres=#select*fromset_dicounts();

-[RECORD1]-------

item_name|macbook

item_price|1200

discount|10

-[RECORD2]-------

item_name|pen

item_price|5

discount|5

-[RECORD3]-------

item_name|fridge

item_price|1000

discount|10

Thefirstthingyounoticeaboutthisfunction,isthatitreturnsaSETOFitemtype.Itusesreturn_nexttobuilduptheresultsetasitprocessesrowsfromtheitemtable.Thefunctionisthenterminatedwithafinalreturnundef(youcanalsousejustreturn).spi_exec_queryexecutesSQLandreturnsthecompleteresultsetasareferencetoanarrayofhashreferences.Ifyouaredealingwithalargenumberofrows,youmaynotwanttousespi_exec_queryasitreturnsalltherowsatonce,butyou’dratherusethespi_queryandspi_fetch_rowfunctions,whichallowyoutoiterateoverthedatalikeacursor.

NoteYoucanreadmoreaboutusingdatainaPL/PerlfunctioninthePostgreSQLdocumentationathttp://www.postgresql.org/docs/current/static/plperl-builtins.html.

http://www.postgresql.org/docs/current/static/plperl-builtins.htm

WritingPL/PerltriggersIfyouwanttowritetriggerfunctionsusingPerl,thenPL/PerlallowsyoutodoallthegoodstuffthatyouhavelearnedsofarusingPL/PgSQL.Let’srewriteanexamplewedemonstratedinChapter5,PL/pgSQLTriggerFunctions,inPL/Perl.Recallthesimple,“Hey,Iamcalled”trigger.ThePL/Perlversionoftheexamplelooksasshowninthefollowingcode.Weprobablydon’tneedtoprovideamorecomplexexample,asthissimpleexampledemonstratesthePL/Perlsyntaxinasufficientway:

CREATEORREPLACEFUNCTIONnotify_trigger_plperl()RETURNSTRIGGER

AS$$

$result=sprintf('Hi,Igot%sinvokedFOR%s%s%son%s',

$_TD->{name},

$_TD->{level},

$_TD->{when},

$_TD->{event},

$_TD->{table_name}

);

if(($_TD->{event}cmp'UPDATE')==0){

$result.=sprintf('OLD=%sANDNEW=%s',$_TD->{old}{i},$_TD->

{new}{i});

$_TD->{new}{i}=$_TD->{old}{i}+$_TD->{new}{i};

elog(NOTICE,$result);

return"MODIFY";

}elsif(($_TD->{event}cmp'DELETE')==0){

elog(NOTICE,"SkippingDelete");

return"SKIP";

}

elog(NOTICE,$result);

$$LANGUAGEplperl;

CREATETABLEnotify_test_plperl(iint);

CREATETRIGGERnotify_insert_plperl_trigger

BEFOREINSERTORUPDATEORDELETEONnotify_test_plperl

FOREACHROW

EXECUTEPROCEDUREnotify_trigger_plperl();

Let’strytoruntheINSERT,UPDATE,andDELETEcommands:

testdb=#INSERTINTOnotify_test_plperlVALUES(1);

NOTICE:Hi,Igotnotify_insert_plperl_triggerinvokedFORROWBEFORE

INSERTonnotify_test_plperl

CONTEXT:PL/Perlfunction"notify_trigger_plperl"

INSERT01

testdb=#UPDATEnotify_test_plperlSETi=10;

NOTICE:Hi,Igotnotify_insert_plperl_triggerinvokedFORROWBEFORE

UPDATEonnotify_test_plperlOLD=1ANDNEW=10


UPDATE1

testdb=#select*fromnotify_test_plperl;

i

----

11

(1row)

postgres=#DELETEFROMnotify_test_plperl;

NOTICE:SkippingDelete


DELETE0

Let’sreviewwhatwehavedonesofar.Wehavecreatedatriggerfunction,atesttable,andanactualtriggerthatrunsbeforeanINSERT,UPDATE,orDELETEforeachrow.

Thetriggerfunctionusesacoupleofthingswehavenotdiscussedsofar.InaPL/Perltriggerfunction,thehashreference$_TDcontainsinformationaboutthecurrenttriggerevent.Youcanseethefulllistofkeysin$_TDathttp://www.postgresql.org/docs/current/static/plperl-triggers.html.

Thetriggerswehaveusedinourexampleareexplainedinthefollowingtable:

Triggername Triggerdescription

$_TD->{name} Thisdenotesthenameofthetrigger.Inourexample,thiswillcontainnotify_trigger_plperl.

$_TD->{level} ThisisaROWorSTATEMENTtrigger.Inourexample,thiswillcontainROW.

$_TD->{when} TheBEFORE,AFTER,orINSTEADOFtrigger.Inourexample,thiswillcontainBEFORE.

$_TD->{event}TheINSERT,UPDATE,DELETE,orTRUNCATEcommand.Inourexample,thiswillcontainINSERT,UPDATE,orDELETE.

$_TD->

{table_name}Thisdenotesthenameofthetable.Inourexample,thiswillcontainnotify_test_plperl.

$_TD->{old}{i} ThiswillcontaintheOLDvalueofthecolumni.Inourexample,thiswillcontain1.

TD->{new}{i} ThiswillcontaintheNEWvalueofthecolumni.Inourexample,thiswillcontain10.

Wehavealsousedautilityfunctioncalledelog()inthetriggerfunction.Thisfunctionemitslogmessages.ThelevelvaluesthatarepossibleareDEBUG,LOG,INFO,NOTICE,WARNING,andERROR.TheERRORvaluepropagatesanerrortothecallingqueryandissimilartoaPerldiecommand.

NoteYoucanviewalltheavailablebuilt-insandutilityfunctionsavailableinPL/Perlathttp://www.postgresql.org/docs/current/static/plperl-builtins.html.

Ourtriggerfunctionreturnsthespecialvalues"MODIFY"and"SKIP"inthecaseofUPDATEandDELETE,respectively.IfaPL/Perltriggerfunctionreturns"MODIFY",itmeansthattheNEWvaluehasbeenmodifiedbythetriggerfunction.IfatriggerfunctionmodifiesaNEWvaluebutdoesnotreturn"MODIFY",thenthechangedonebythefunctionwillbediscarded.The"SKIP"impliesthattheoperationshouldnotbeexecuted.

http://www.postgresql.org/docs/current/static/plperl-triggers.html

http://www.postgresql.org/docs/current/static/plperl-builtins.html

UntrustedPerlWediscusseduntrustedPL/PythonUinChapter8,UsingUnrestrictedLanguages.PL/Perlisalsoavailableasanuntrustedlanguage.Thetrustedversionrunsinsideasecuritycontextthatdoesnotallowinteractionwiththeenvironment.JustlikePL/Pythonu,wecanbypassthesecurityrestrictionsusingPL/Perluortheuntrustedversion.Let’srewritethedirectorylistingfunctionlist_folderfromChapter8,ListingdirectorycontentstoaPerlequivalent:

CREATEORREPLACEFUNCTIONlist_folder_plperl(directoryVARCHAR)RETURNS

SETOFTEXT

AS$$

my$d=shift;

opendir(D,"$d")||elog(ERROR,'Cantopendirectory'.$d);

my@list=readdir(D);

closedir(D);

foreachmy$f(@list){

return_next($f);

}

returnundef;

$$LANGUAGEplperlu;

Let’srunourfunction,asshownhere:

testdb=#SELECTlist_folder_plperl('/usr/local/pgsql/bin');

list_folder_plperl

--------------------

.

..

clusterdb

createdb

createlang

createuser

dropdb

droplang

dropuser

ecpg

initdb

pg_basebackup

pg_config

pg_controldata

pg_ctl

pg_dump

pg_dumpall

pg_isready

pg_receivexlog

pg_resetxlog

pg_restore

postgres

postmaster

psql

reindexdb

vacuumdb

(26rows)

Ifwetrytocreatetheprecedingfunctionasplperlinsteadofplperlu,wewillgetanerrorsuchasERROR:'opendir'trappedbyoperationmaskatline3bythevalidator,becausewearetryingtoaccessthehostsystem.

SummaryThepowerfulPerllanguageisavailableinPostgreSQLasPL/PerlorPL/Perlu.ThisallowsyoutowritestoredproceduresinPerlandtotakeadvantageofallthecoolfeaturesthatPerlhastooffer,suchasaverylargecollectionofmodulesavailableonCPAN.YoucandoalmosteverythingyouwantwithPL/pgSQLorPL/Python,includingdatabaseaccessandwritingtriggers.TheuntrustedversionofPL/Perlallowsyoutointeractwiththeenvironment.PL/PerlwillnormallyoutperformPL/pgSQLinnon-dataintensivefunctionsthatfocusmoreonstringmanipulationandcomputation.

Inthenextchapter,wewilldiscussanotherpopularPLlanguagecalledPl/Tcl.

Chapter12.PL/Tcl–TclProceduralLanguageToolsCommandLanguage(Tcl),alsocommonlyknownastickle,hasbeenaroundforalongtime.ItwascreatedbyJohnOusterhoutin1988andgotalotoftractionforrapidprototypingandscriptedapplications.

Inthischapter,wewilltakeabrieflookatPL/Tcl.BetweenPL/Perl,PL/Python,andPL/pgSQL,youhaveverypowerfullanguagesavailableatyourdisposalthatcandoalmostanythingyouneed.Forsomeotherthings,youhavetheoptiontowriteyourfunctionsinC.YoumightwonderwhyitisusefultodiscussPL/Tcl.Foralongtime,intheearlydaysofPostgreSQL,PL/Tclu(untrustedPL/Tcl)wastheonlywaytodothingsoutsidePostgreSQL,suchasinteractingwiththeoperatingsystem.Alotofpeoplestilluseit,andIpersonallythinkthatitissocleanandpowerfulthatitshouldnotbeoverlooked.

PL/Tclisavailableasatrustedandanuntrustedlanguage.ThisisachievedbyprovidingtwodifferentTclinterpreters.PL/TcluusesthestandardTclinterpreter,whilePL/TclusesaspecialSafeTclmechanism.

NoteYoucanreadmoreaboutsafeTclathttp://www.tcl.tk/software/plugin/safetcl.html.


InstallingPL/TclandwritingasimplefunctionPassingsimpleandcomplexparametersAccessingadatabasefromaPl/TclfunctionWritingdatabasetriggersusingPl/Tcl

http://www.tcl.tk/software/plugin/safetcl.html

InstallingPL/TclPL/Tclisnotinstalledbydefault,ifyou’veusedthestandardsourcedistributiontoinstallPostgreSQL.IfyoucompiledPostgreSQLfromthesource,youneedtoruntheconfigurescriptwiththe–-with-tcloption.

Ifyou’veusedabinarydistributiononyourplatform,youcannormallyinstallPL/Tclusingyourpackagemanager.Youcansearchforpostgresql-pltcl,orasimilarpackagename,asitdiffersacrossdistributions.OncePostgreSQLiscompiledwiththecorrectoption,oryouhaveinstalledtheappropriatepackage,youcancreatethelanguageusingthecreatelangutilityortheCREATELANGUAGEcommand:

$createlangpltcltemplate1

Youcanusealsousetheuntrustedversiontocreatethelanguage,asfollows:

$createlangpltclutemplate1

AsimplePL/TclfunctionNow,let’swriteourfirstsimpleTclfunctiontomakesurethatPL/Tclisinstalled.Wewillwriteasimplefactorialcalculationfunction,asshownhere:

CREATEORREPLACEFUNCTIONtcl_factorial(integer)RETURNSinteger

AS$$

seti1;setfact1

while{$i<=$1}{

setfact[expr$fact*$i]

incri

}

return$fact

$$LANGUAGEpltclSTRICT;

Thisfunctioncalculatesthefactorialofanumberinaniterativeway.Let’stryandrunit:

postgres=#SELECTtcl_factorial(5);

tcl_factorial

---------------

120

(1row)

ItworksandthefunctionlookssimilartootherfunctionswehavebeenwritinginPL/pgSQLandPL/Python.TheCREATEFUNCTIONstatementcreatesafunction.Itneedsaname,functionargumenttypelist(youhavetouseparentheses,eveniftherearenoarguments),aresulttype,andalanguage.

ThebodyofthefunctionisjustaTclscript.PostgreSQLpassesthebodyontoaTclinterpretertorunthissubroutineandreturntheresults.Thefunctionargumentsarepassedontothescriptas$1,$2,…$n.

NullcheckingwithStrictfunctionsTheSTRICTkeywordwillsaveusfromcheckingthenullinputparameters.IfyouhavespecifiedafunctionasSTRICTandanyoftheinputparametersarenull,itresultsinthefunctionnotbeingcalledandanullresultsetisreturnedimmediately:

postgres=#SELECTtcl_factorial(null);

tcl_factorial

---------------

(1row)

Ifyoudon’twanttocreateaSTRICTfunction,oryou’dliketodothenullcheckingyourself,youcanrewritethefunction,asshowninthefollowingcodesnippet.Thisisusefulifyouhavemultipleparametersandyouwanttoallowsomeparameterstobenull:

CREATEORREPLACEFUNCTIONtcl_factorial_ns(integer)RETURNSinteger

AS$$

if{[argisnull1]}{

elogNOTICE"inputisnull"

return-1

}

seti1;setfact1

while{$i<=$1}{

setfact[expr$fact*$i]

incri

}

return$fact

$$LANGUAGEpltcl;

Theargisnullfunctionisusedtocheckfornullvalues.Intheprecedingexample,thefunctionreturns-1iftheinputargumentisnull,justtodemonstratethatitworks.Ifyouwanttoreturnanullvaluefromthefunction,youcanusethebuilt-infunctionreturn_null.Intheprecedingexample,youcanalsoseehowtousetheelogfunctioninPL/Tcl:

postgres=#SELECTtcl_factorial_ns(null);

tcl_factorial_ns

------------------

-1

(1row)

TheparameterformatAllinputparameterspassedtoPL/Tclareconvertedtotext.WithinaPL/Tclfunction,allvaluesaretext.Whenthefunctionreturns,anotherconversionisperformedfromthetextstringtothereturntypeofthefunction,aslongasthetextbeingreturnedisanappropriaterepresentationofthereturntypeofthePl/Tclfunction;otherwise,thefunctionwillresultinanerror.

PassingandreturningarraysIfyoupassarraytypesasanargumenttothePL/Tclfunction,theyarepassedasastringvalue,alongwiththebracketsandthecommas.Let’stakealookatanexample:

CREATEORREPLACEFUNCTIONtcl_array_test(integer[])RETURNSint

AS$$

setlength[stringlength$1]

return$length

$$LANGUAGEpltcl;

testdb=#selecttcl_array_test(ARRAY[1,2,3]);

tcl_array_test

----------------

7

(1row)

Youareprobablysurprisedatthereturnvalueoftheprecedingfunction.Youpassedanintegerarraytothefunctionthatisconvertedtoastringvalue{1,2,3},thelengthofwhichisindeed7.Ifyouwanttoprocessarrayvaluesindependently,youneedabitofstringmanipulationtoextractthelistoutofthestring,dothemanipulation,andconvertitbacktothestringformatthatyoureceiveditin.

Let’stakealookatanexamplePL/Tclfunctionthatwillreverseanintegerarrayandreturnthereversedintegerarray:

CREATEORREPLACEFUNCTIONtcl_reverse_array(integer[])RETURNSinteger[]

AS$$

setlst[regexp-all-inline{[0-9]}$1]

setlst[join[lreverse$lst]","]

setlst"{$lst}"

return$lst

$$LANGUAGEpltcl;

postgres=#selecttcl_reverse_array(ARRAY[1,2,3]);

tcl_reverse_array

-------------------

{3,2,1}

(1row)

Theprecedingfunctiondoesthefollowing:

1. Thetcl_reverse_arrayfunctioncleansuptheinputparameterbycreatingalistoutofthestringandremovingthe{and,characters.Thisisdoneusingaregularexpressionandbyonlyextractingnumericvaluesoutofthestring.

2. Itusesthelreversefunctiontoreversethecontentsofthelistandthenjointheelementsofthelistbackasanarray,usingthejoinfunctionandusing,asthejoincharacter.

3. Then,itaddsbracketstothestringbeforereturningit.Thereturnstringisconvertedtoanintegerarrayasitisreturnedbythefunction.

Passingcomposite-typeargumentsAcomposite-type,suchasatable,orauser-definedtypeispassedtothePL/Tclfunctionasanassociativearray(Hashtable).Theattributenamesofthecomposite-typearetheelementnamesinthearray.AttributeswithNULLvaluesarenotavailableintheTclarray.

Let’stakealookatanarbitraryexample.Wewillcreateafunctionthattakesacompositetypeasanargumentanddoessomecalculationsbasedontheattributevaluesofthetype.Let’screateourtypeandthefunction:

CREATETABLEorders(orderidint,num_peopleinteger,order_amountdecimal);

INSERTINTOordersVALUES(1,1,23);


INSERTINTOordersVALUES(3,5,567.25);


CREATEORREPLACEFUNCTIONtip_calculator(orders,integer)RETURNSdecimal

AS$$

if{$1(order_amount)>0}{

settip[expr(double($1(order_amount)*$2)/100)/$1(num_people)]

settip[format"%.2f"$tip]

return$tip

}

return0;

$$LANGUAGEpltcl;

Theorderstableisquitesimpletounderstand:itcontainsanorderID,thenumberofpeopleintheorderstable(num_people),andthetotalcostoftheorder(order_amount).

Thefunctioncalculatesthetipperperson,basedonthepriceofthemeal(order_amount)andnumberofpeople.Ittakesanargumentofthetypeordersandanintegervaluethatrepresentthepercentageofthetipthatshouldbepaid.

Thefunctionbodycalculatesthetipandformatstheresulttotwodecimalplaces.Ifwerunourfunction,thisiswhatweshouldsee:

postgres=#SELECTtip_calculator(orders.*,5)AS"tipperperson"FROM

orders;

tipperperson

----------------

1.15

2.62

5.67

5.00

(4rows)

Youcanseethatalltheattributesoftheorderstypecanbeaccessedinthefunctionas$1(order_amount),$1(num_people),andsoon.Thisfunctiongetscalledonceforeachrowoftheorderstableandcalculatestheamountofthetiptobepaidperpersonforeachorder,accordingtovariousparameters.

Note

YoucanseethefulllistofassociativearraycommandsintheofficialdocumentationforTclathttp://www.tcl.tk/man/tcl8.5/tutorial/Tcl22.html.

Atthismoment,PL/Tclfunctionsdon’tallowthereturningofaSETOFtype,orcompositetype,fromafunction.

http://www.tcl.tk/man/tcl8.5/tutorial/Tcl22.html

AccessingdatabasesPL/TclfunctionsprovideyouwithSPIfunctionstoaccessthedatabaseandrunDML/DDLstatements.

Thefunctionsarethefollowing:

spi_exec:ThisexecutesaSQLstatementspi_prepare:ThispreparesaSQLstatementspi_execp:Thisexecutesapreparedstatement

Thespi_execfunctionhasthefollowingsyntax:

spi_exec?-countn??-arrayname?command?loop-body?

Thespi_execfunctionrunsaSQLstatement,andittakessomeoptionalparameters,asfollows:

-count:Thisparameterallowsyoutospecifythemaximumnumberofrowsprocessedbythecommand.Ifyouprovidethevalue3,only3rowswillbeprocessed.ThisissimilartospecifyingFETCH[n]inacursor.-array:Ifthisparameterisspecified,thecolumnvaluesarestoredintoanamedassociativearrayandthecolumnnamesareusedasarrayindexes.Ifthisparameterisnotspecified,theresultvaluesarestoredintheTclvariablesofthesamename.

Ifthereisaloopbodyspecified,thenitistreatedasascriptthatisrunforeachrow.

Let’stakealookatanexampleofhowtorunSQLstatementsinsideaPL/Tclfunction.Thefollowingexample,createsafunctionthatloopsovertherowsinatableandupdatesacolumnineachrow:

CREATETABLEemp_sales(empidintPRIMARYKEY,sales_amntdecimal,

comm_percdecimal,

comm_amntdecimal);

INSERTINTOemp_salesVALUES(1,32000,5,NULL);

INSERTINTOemp_salesVALUES(2,5231.23,3,NULL);

INSERTINTOemp_salesVALUES(3,64890,7.5,NULL);

CREATEORREPLACEFUNCTIONtcl_calc_comm()RETURNSint

AS$$

spi_exec-arrayC"SELECT*FROMemp_sales"{

setcamnt[expr($C(sales_amnt)*$C(comm_perc))/100]

spi_exec"updateemp_sales

setcomm_amnt=[format"%.2f"$camnt]

whereempid=$C(empid)"

}

$$LANGUAGEpltcl;

Theprecedingexample,doesthefollowing:

Itcreatesatableemp_sales,whichcontainstheemployeeID,howmuchsalestheemployeehasmade,andwhatistheircommissionpercentage.Thelastcolumnthat

representsthetotalcommissiontobepaidtotheemployeebasedonhis/hersalesandhis/hercommissionpercentageisleftblankintentionally,anditisfilledbyourfunction.Itfillsthetablewithsomerandomdata.Then,thefunctionbodyrunsaSQLstatementusingthespi_execcommand,andthecolumnvaluesarereturnedintheassociativearraycalledC.Finally,theloopbodycalculatesthecommissionvalueandupdatesthecomm_amntcolumnforeachrow.

NoteYoucanreadmoreaboutaccessingthedatabaseinaPL/Tclfunctionathttp://www.postgresql.org/docs/current/interactive/pltcl-dbaccess.html.

http://www.postgresql.org/docs/current/interactive/pltcl-dbaccess.htm

WritingPL/TcltriggersIfyouwanttowritetriggerfunctionsusingTcl,thenPL/PTclallowsyoutodoallthegoodstuffthatyouhavelearnedsofar,usingPL/pgSQL,PL/Perl,andPL/Python.Let’srewriteanexamplewedemonstratedinChapter5,PL/pgSQLTriggerFunctions.Recallthesimple,“Hey,Iamcalled”trigger.ThisishowthePL/Tclversionoftheexamplelooks:

CREATEORREPLACEFUNCTIONnotify_trigger_pltcl()RETURNSTRIGGER

AS$$

setresult[format"Hi,Igot%sinvokedFOR%s%s%son%s"$TG_name

$TG_level$TG_when$TG_op$TG_table_name]

if{$TG_op=="UPDATE"}{

appendresult[format"OLD=%sANDNEW=%s"$OLD(i)$NEW(i)]

setNEW(i)[expr$OLD(i)+$NEW(i)]

elogNOTICE$result

return[arraygetNEW]

}elseif{$TG_op=="DELETE"}{

elogNOTICE"DELETE"

returnSKIP

}elogNOTICE$result

returnOK

$$LANGUAGEpltcl;

CREATETABLEnotify_test_pltcl(iint);

CREATETRIGGERnotify_insert_pltcl_trigger

BEFOREINSERTORUPDATEORDELETEONnotify_test_pltcl

FOREACHROW

EXECUTEPROCEDUREnotify_trigger_pltcl();

TheprecedingcodeisalmostacarboncopyoftheoneinChapter11,PL/Perl–PerlProceduralLanguage,butintheTclsyntax.ItdemonstrateshowtouseTGvariables,howtomodifyNEWvalues,andhowtoSKIPanoperation.

Itprints“HeyIgotinvoked”withdifferentattributesofthetrigger.InthecaseofanUPDATE,italsoprintstheNEWandOLDvalues,aswellasmodifiestheNEWvalueandskipstheDELETEoperation.APL/Tcltriggerfunctioncanreturnthefollowingvalues:

OK:ThisisthedefaultvalueandimpliesthattheoperationexecutedbytheuserwillproceednormallySKIP:Thisreturnvalueimpliesthattheuser-executedoperationwillbesilentlyignoredLIST:ThisisreturnedbyarraygetandimpliesthatamodifiedversionoftheNEWarrayshouldbereturnedOLDandNEW:ThesevaluesareavailableasassociativearraysinaPL/Tcltriggerfunction,andtheattributesofthetablearethenamesoftheelementsinthearray

Let’srunINSERT,UPDATE,andDELETEtoseetheresults:

postgres=#INSERTINTOnotify_test_pltclVALUES(1);

NOTICE:Hi,Igotnotify_insert_pltcl_triggerinvokedFORROWBEFORE

INSERTonnotify_test_pltcl

INSERT01

postgres=#UPDATEnotify_test_pltclSETi=10;

NOTICE:Hi,Igotnotify_insert_pltcl_triggerinvokedFORROWBEFORE

UPDATEonnotify_test_pltclOLD=1ANDNEW=10

UPDATE1

postgres=#DELETEFROMnotify_test_pltcl;

NOTICE:DELETE

DELETE0

postgres=#SELECT*FROMnotify_test_pltcl;

i

----

11

(1row)

InaPL/Tcltriggerfunction,the$TGvariablescontaininformationaboutthecurrenttriggerevent.Youcanseethefulllistofvariablesathttp://www.postgresql.org/docs/current/interactive/pltcl-trigger.html.

Thevariableswehaveusedinourexampleareexplainedinthefollowingtable:

Triggername Description

$TG_name Thisdenotesthenameofthetrigger.Inourexample,thiswillcontainnotify_trigger_pltcl.

$TG_level ThisisaROWorSTATEMENTtrigger.Inourexample,thiswillcontainROW.

$TG_when ThisisaBEFORE,AFTER,orINSTEADOFtrigger.Inourexample,thiswillcontainBEFORE.

$TG_opThisisanINSERT,UPDATE,DELETE,orTRUNCATEtrigger.Inourexample,thiswillcontainINSERT,UPDATE,orDELETE.

$TG_table_name Thisdenotesthenameofthetable.Inourexample,thiswillcontainnotify_test_pltcl.

$OLD(i) ThiswillcontaintheOLDvalueofthecolumni.Inourexample,thiswillcontain1.

$NEW(i) ThiswillcontaintheNEWvalueofthecolumni.Inourexample,thiswillcontain10.

http://www.postgresql.org/docs/current/interactive/pltcl-trigger.html

UntrustedTclUsinguntrustedTcl(pltclu)isoneoftheoldestwaystodothingsoutsidethedatabase.pltcluisexecutedusinganormalTclinterpreterandisprettymuchfreetodoanythingyou’dlike.Tclhasalotofcommandsavailabletointeractwiththeoperatingsystemandtheenvironment.Wewillnowtakealookatafewsimpleexamples.

Thefirstexample,readsthecontentsofthefileandreturnsthemastext,asshowninthefollowingcode:

CREATEORREPLACEFUNCTIONread_file(text)RETURNStext

AS$$

setfptr[open$1]

setfile_data[read$fptr]

close$fptr#closethefileasitisalreadyread

return$file_data

$$LANGUAGEpltclu;

Thefunctionisquitesimple;itopensafileprovidedasaparameter,readsallitscontentsatonce,andreturnsthetext.

Let’sruntheprecedingfunction:

postgres=#selectread_file('/usr/local/pgsql/data/postmaster.pid');

read_file

-----------------------

61588+

/usr/local/pgsql/data+

1401041744+

5432+

/tmp+

localhost+

5432001196608+

(1row)

Hereisanotherfunctionthatdoesadirectorylisting,similartotheplperluexampleinChapter11,PL/Perl–PerlProceduralLanguage.SincePL/TcldoesnotsupportthereturningoftheSETOFtext,wewillsimplyreturnthecompletedirectorylistingasonestring.Asyoucanseeinthefollowingcode,thiscanbedonewithasinglelineinTcl:

CREATEORREPLACEFUNCTIONlist_directory(text)RETURNStext

AS$$

setdirList[glob-nocomplain-directory$1*.*]

return$dirList

$$LANGUAGEpltclu;

testdb=#SELECTlist_directory('/tmp');

list_directory

---------------------------------------------

/tmp/lu84iont.tmp/tmp/unity_support_test.0

(1row)

WereadthecontentsofthefileusingTcl’sglobfunctionandjustreturnedthecontentsasastring.

Let’stakealookatonelastPL/Tclufunctionthatwillexportthecontentstoatableasa.csvfile:

CREATEORREPLACEFUNCTIONdump_table(text,text)RETURNSint

AS$$

setfilename$1

setfileId[open$filename"w"]

spi_exec-arrayemp"SELECT*FROM$2"{

setrow[format"%d,%.2f,%.2f,%.2f"$emp(empid)$emp(sales_amnt)

$emp(comm_perc)$emp(comm_amnt)]

puts$fileId$row

}

close$fileId

return0;

$$LANGUAGEpltcluSTRICT;

Again,thisfunctionisquitesimpleandusestheconceptswehavediscussedbefore.Ititeratesoverthetableprovidedasthesecondparametertothisfunctionandstoresthedatainafilenamed,asperthefirstparameterofthisfunction.Thefileiswrittenlinebylineascomma-separatedvaluesbyiteratingoverthetabledatausingspi_exec.

Let’srunthisfunctionontheemp_salestablethatwecreatedearlierinthischapter:

postgres=#selectdump_table('emp.txt','emp_sales');

dump_table

------------

0

(1row)

Thisseemstowork.Wecanverifythisbyrunningtheread_filefunctionwewroteearlier:

postgres=#selectread_file('emp.txt');

read_file

-------------------------

1,32000.00,5.00,1600.00+

2,5231.23,3.00,156.94+

3,64890.00,7.50,4866.75+

(1row)

Itseemsthatthefunctionworks,andwehaveaCSVdumpofthetableinnotime.

SummaryThepowerful,yetclean,TcllanguageisavailableinPostgreSQLasPL/TclandtheuntrustedPL/Tclu.BoththelanguagesusedifferentTclinterpreters.PL/TcluistheoldestlanguageusedinPostgreSQL,toaccessthingsoutsidethedatabase.ItallowsyoutowritestoredproceduresinTclandtakesadvantageofallthecoolfeaturesthatTclhastooffer.YoucandoalmosteverythingyoucanwithotherPLlanguages,includingdatabaseaccessandwritingtriggers.TheonlymajordisadvantageofPL/Tclisthatitdoesnotallowyoutoreturncompositetypesandsetsfromafunction.

Inthenextchapter,wewillexplorehowtopublishyourcodeasaPostgreSQLextension.

Chapter13.PublishingYourCodeasPostgreSQLExtensionsIfyouarenewtoPostgreSQL,nowisthetimetodanceforjoy.

Nowthatyou’redonedancing,I’lltellyouwhy.Youhavemanagedtoavoidthe“badolddays”ofcontribmodules.ContribmodulesaretheinstallationsystemsthatwereusedtoinstallrelatedPostgreSQLobjectspriortoVersion9.1.Theymaybeadditionaldatatypes,enhancedmanagementfunctions,orjustreallyanytypeofmoduleyouwanttoaddtoPostgreSQL.Theyconsistofanygroupofrelatedfunctions,views,tables,operators,types,andindexesthatwerelumpedintoaninstallationfileandcommittedtothedatabaseinonefellswoop.Unfortunately,contribmodulesonlyprovidedforinstallation,andnothingelse.Infact,theywerenotreallyaninstallationsystematall.TheywerejustsomeunrelatedSQLscriptsthathappenedtoinstalleverythingthattheauthorthoughtyouneeded.

PostgreSQLextensionsprovidemanynewservicesthatapackagemanagementsystemshouldhave.Well,atleasttheonesthatmoduleauthorscomplainedthemostaboutnotbeingpresent.

Someofthenewfeaturesthatyouwillbeintroducedtointhischapterinclude:

VersioningDependenciesUpdatesRemoval

WhentocreateanextensionWell,firstyouhavetounderstandthatextensionsareallabouttogetherness.Oncetheobjectsfromacontribmodulewereinstalled,PostgreSQLprovidednowaytoshowarelationshipbetweenthem.Thisledmanydeveloperstocreatetheirown(andattimesratheringenious)methodstoversion,update,upgrade,anduninstallallofthenecessary“stuff”togetafeaturetowork.

So,thefirstquestiontoaskyourselfwhencontemplatingaPostgreSQLextensionasawaytopublishyourcodeis“Howdoesallofthe‘stuff’inmyextensionrelatetogether?”

Thisquestionwillhelpyoumakeextensionsthatareasgranularasreasonable.IftheobjectiveistoenhancePostgreSQLwiththeabilitytoprovideaninventorymanagementsystem,itmightbebettertostartwithanextensionthatprovidesabillofmaterialsdatatypefirst,andsubsequentlybuildadditionalextensionsthataredependentuponthatone.Themoralofthestoryistodreambig,butcreateeachextensionwithonlythesmallestnumberofrelateditemsthatmakesense.

AgoodexampleofanextensionthatprovidesafeaturetoPostgreSQLisOpenFTS.ThisextensionprovidesfulltextsearchingcapabilitiestoPostgreSQLbycreatingdatatypes,indexes,andfunctionsthatarewellrelatedtoeachother.

AnothertypeofextensionisPostGIS,whichprovidesarichsetoftoolstodealwithgeographicinformationsystems.AlthoughthisextensionprovidesmanymorebitsoffunctionalitythanOpenFTS,itisstillasgranularaspossiblebyvirtueofthefactthateverythingthatisprovidedisnecessaryforgeographicsoftwaredevelopment.

Possibly,youareabookauthor,andtheonlyrelationshipthattheobjectsinyourextensionhaveisthattheyneedtobeconvenientlyremovedwhenyourpoorvictim…ahem…thereaderisthroughwiththem.Welcometothewondersofextensions.

Foralistofveryusefulextensionsthathavegainedsomecommunitypopularity,youmightwanttotakealookatthispagefairlyoften:http://www.postgresql.org/download/products/6/.

YoushouldalsotakealookatthePostgreSQLextensionnetworkathttp://www.pgxn.org.Pleasenotethatinstallingextensionsgenerallymeansrunningascriptasasuperuser;andnearlyanyonecanuploadtopgxn,meaningthatindividualsreallyneedtovetanythingtheygetfrompgxnverycarefully.

NoteTofindoutwhatobjectscanbepackagedintoanextension,lookattheALTEREXTENSIONADDcommandinthePostgreSQLdocumentation:

http://www.postgresql.org/docs/current/static/sql-alterextension.html

http://www.postgresql.org/download/products/6/

http://www.pgxn.org

http://www.postgresql.org/docs/current/static/sql-alterextension.html

UnpackagedextensionsStartingwithVersion9.1,PostgreSQLprovidesaconvenientwaytomovefromtheprimordialoozeofunversionedcontribmodulesintothebravenewworldofextensions.Basically,youprovideaSQLfiletoshowtherelationshipoftheobjectstotheextension.Thecontribmodule’scubeprovidesagoodexampleofthisincube--unpackaged--1.0.sql:

/*contrib/cube/cube--unpackaged--1.0.sql*/

—complainifscriptissourcedinpsql,ratherthanviaCREATEEXTENSION

\echoUse"CREATEEXTENSIONcube"toloadthisfile.\quit

ALTEREXTENSIONcubeADDtypecube;

ALTEREXTENSIONcubeADDfunctioncube_in(cstring);

ALTEREXTENSIONcubeADDfunctioncube(doubleprecision[],double

precision[]);

ALTEREXTENSIONcubeADDfunctioncube(doubleprecision[]);

...

ThecodethatprovidesmultidimensionalcubesforPostgreSQLhasbeenstableforquitesometime.Itisunlikelythatanewversionwillbecreatedanytimesoon.Theonlyreasonforthismoduletobeconvertedintoanextensionistoallowforeasyinstallationandremoval.

Youwouldthenexecutethecommand:

CREATEEXTENSIONcubeFROMunpackaged;

Theunrelateditemsarenowgroupedtogetherintotheextensionnamedcube.Thisalsomakesiteasierforthepackagingmaintaineronanyplatformtoincludeyourextensionintotherepository.We’llshowyouhowtomakethepackagestoinstallyourextensionintheBuildinganextensionsection.

ExtensionversionsTheversionmechanismforPostgreSQLextensionsissimple.Nameitwhateveryouwantandgiveitwhateveralphanumericversionnumberthatsuitsyourfancy.Easy,eh?Justnamethefilesinthisformat:

extension--version.sql

Ifyouwanttoprovideanupgradepathfromoneversionofyourextensiontoanother,youwouldprovidethefile:

extension--oldversion--newversion.sql

ThissimplemechanismallowsPostgreSQLtoupdateanextensionthatisalreadyinplace.Gonearethedaysofpainfulexportingandre-importingdatajusttochangethedefinitionofadatatype.So,let’sgoaheadandupdateourexampleextensionusingthefilepostal--1.0--1.1.sql.Thisupdateisaseasyas:

ALTEREXTENSIONpostalUPDATETO'1.1';

NoteAnoteofcaution:PostgreSQLdoesnothaveanyconceptofwhatyourversionnumbermeans.Inthisexample,theextensionwasupdatedfromVersion1.0to1.1becauseweexplicitlyprovidedascriptforthatspecificconversion.PostgreSQLdidnotdeducethat1.1follows1.0.Wecouldhavejustaseasilyusedthenamesoffruitsorhistoricalbattleshipsforourversionnumbersandtheresultwouldhavebeenthesame.

PostgreSQLwillusemultipleupdatefilesifnecessarytoachievethedesiredresult.Giventhefollowingcommand:

ALTEREXTENSIONpostalUPDATETO'1.4';

PostgreSQLwillapplythefilespostal--1.1--1.2.sql,postal--1.2--1.3.sqlandpostal--1.3--1.4.sqlinthecorrectordertoachievethedesiredversion.

Youmayalsousethistechniquetoprovideupgradescriptsthatareinfactdowngradescripts,thatis,theyactuallyremovefunctionality.Becarefulwiththisthough.Ifapathtoadesiredversionistodowngradebeforeanupgrade,PostgreSQLwilltaketheshortestroute.Thismayresultinsomeunintendedresults,includingdataloss.Myadvicewouldbetonotprovidedowngradescripts.Theriskjustisn’tworthit.

The.controlfileAlongwiththeextensioninstallationscriptfile,youmustprovidea.controlfile.The.controlfileforourexamplepostal.controllookslikethis:

#postaladdressprocessingextension

comment='utilitiesforpostalprocessing'

default_version='1.0'

module_pathname='$libdir/postal'

relocatable=true

requires='plpgsql'

Thepurposeofthe.controlfileistoprovideadescriptionofyourextension.Thismetadatamayincludedirectory,default_version,comment,encoding,module_pathname,requires,superuser,relocatable,andschema.

ThemainPostgreSQLdocumentationforthisfileislocatedathttp://www.postgresql.org/docs/current/static/extend-extensions.html.

Thisexampleshowstherequiresconfigurationparameter.OurextensiondependsontheprocedurallanguagePL/pgSQL.Onmostplatforms,itisinstalledbydefault.Unfortunately,itisnotinstalledonallplatforms,andnothingshouldbetakenforgranted.

Multipledependenciescanbeindicatedbyseparatingthemwithcommas.Thiscomesinveryhandywhenconstructingasetofservicesbasedonmultipleextensions.

Aswementionedintheprevioussection,PostgreSQLdoesnotprovideanyinterpretationoftheversionnumberofanextension.Versionscanbenamesaswellasnumbers,sothereisnowayforPostgreSQLtointerpretthatpostal--lamb.sqlcomesbeforepostal--sheep.sql.Thisdesignlimitationposesaproblemtotheextensiondeveloper,inthatthereisnowaytospecifythatyourextensiondependsonaspecificversionofanotherextension.Iwouldlovetoseethisconfigurationparameterenhancedwithasyntaxlikerequires=postgis>=1.3,butalas,nosuchconstructionexistsatthemoment.

http://www.postgresql.org/docs/current/static/extend-extensions.html

BuildinganextensionWehavealreadycoveredthebasicsofcreatingascriptfileanda.controlfile.Actually,thatisallthatisnecessaryforaPostgreSQLextension.Youmaysimplycopythesefilesintothesharedextensiondirectoryonyourcomputerandexecutethefollowingcommand:

CREATEEXTENSIONpostal;

Thiswillinstallyourextensionintothecurrentlyselecteddatabase.

ThesharedextensionpathisdependentonhowPostgreSQLisinstalled,butforUbuntu,itis/usr/share/postgresql/9.2/extension.

However,thereisamuchbetterwaytodothisthatworkswithanypackagemanageronanyplatform.

PostgreSQLprovidesanextension-buildingtoolkitasapartoftheserverdevelopmentpackage.ToinstallthispackageonUbuntu,youcantype:

sudoapt-getinstallpostgresql-dev-9.4

ThiswillinstallallofthePostgreSQLsourcecodenecessarytocreateandinstallanextension.YouwouldthencreateafilenamedMakefileinthesamedirectoryastherestofyourextensionfiles.Thecontentofthisfilelookslikethis:

EXTENSION=postal

DATA=postal--1.0.sql

PG_CONFIG=pg_config

PGXS:=$(shell$(PG_CONFIG)--pgxs)

include$(PGXS)

ThissimpleMakefilefilewillcopyyourextensionscriptfileandthe.controlfileintothepropersharedextensiondirectoryonanyplatform.Invokeitwiththiscommand:

sudomakeinstall

Youwillseesomeoutputlikethis:

/bin/mkdir-p'/usr/share/postgresql/9.4/extension'

/bin/sh


m644./postal.control'/usr/share/postgresql/9.4/extension/'

/bin/sh


m644./postal--1.0.sql'/usr/share/postgresql/9.4/extension/'

Yourextensionisnowlocatedintheproperdirectoryforinstallation.Youcaninstallitintothecurrentdatabasewith:

CREATEEXTENSIONpostal;

Youwillthenseetheconfirmationtextlettingyouknowthatyouhavenowgonepostal:

CREATEEXTENSION

InstallinganextensionExtensionsthathavebeenpackagedforyoubyyourfriendlydistributionmanagerareverysimpletoinstallusingthefollowingcommand:

CREATEEXTENSIONextension_name;

MostofthepopularLinuxdistributionsincludeapackagecalledsomethinglikepostgresql-contrib-9.4.ThisnamingconventionisleftoverfromthecontribstyleinstallationofPostgreSQLobjects.Don’tworry,forPostgreSQL9.4,thispackagewillactuallyprovideextensionsratherthancontribmodules.

TofindoutwherethefileswereplacedonUbuntuLinux,youcanexecutethefollowingcommand:

pg_config--sharedir

Thiswillshowyoutheinstallationdirectoryofsharedcomponents:

/usr/share/postgresql/9.4

Theextensionswillbelocatedinadirectorycalled,extension,immediatelybelowtheshareddirectory.Thiswillthenbenamed/usr/share/postgresql/9.2/extension.

Toseewhatextensionsareavailableforyoutoinstall,trythiscommand:

ls$(pg_config–sharedir)/extension/*.control

Youcanalsoseethelistofinstalledextensionsusingtheplsqlmeta-command\dx.

ThiswillshowyoualltheextensionsthathavebeenmadeavailabletoyoubyyourLinuxdistribution’spackagemanagementsystem.

Forextensionsthatyouhavecreatedyourself,youmustcopyyourSQLscriptfileandthe.controlfiletothesharedextensiondirectorybeforeinvokingCREATEEXTENSIONinPostgreSQL.

cppostal.controlpostal--1.0.sql$(pg_config--sharedir)/extension

Toseetheprocedurefordoingthisreliablyonanytargetplatform,refertotheBuildinganExtensionsection.

ViewingextensionsQueryingthepg_extensionsystemvieworusingthemeta-command\dxwillshowtheextensionscurrentlyinstalledinthedatabase.

postgres=#\dx

Listofinstalledextensions

-[RECORD1]-----------------------------

Name|pgcrypto

Version|1.0

Schema|public

Description|cryptographicfunctions

-[RECORD2]-----------------------------

Name|plperl

Version|1.0

Schema|pg_catalog

Description|PL/Perlprocedurallanguage

-[RECORD3]-----------------------------

Name|plpgsql

Version|1.0

Schema|pg_catalog

Description|PL/pgSQLprocedurallanguage

Theextensionsthatarereadytobeinstalledcanbeviewedfromthepg_available_extensionsorpg_available_extension_versionssystemviews.

PublishingyourextensionThankyouforcontributingtothePostgreSQLcommunity!Yoursupportwillnotgounnoticedinthisgatheringoflike-mindedindividualswhoareallslightlysmarterthaneachother.Yourworkwillbeseenbydozensofdeveloperslookingforcommunitysolutionstocommonproblems.Youhaveindeedmadetheopensourceworldabetterplace.

Sincewearetalkingaboutpublication,youshouldconsiderthelicensingmodelforyourextension.Thepublicationmethodsthatweareabouttodescribeassumethattheextensionwillbemadeavailabletothegeneralpublic.Assuch,pleaseconsiderthePostgreSQLlicenseforyourextension.Youcanfindthecurrentonehere:

http://www.postgresql.org/about/licence/

http://www.postgresql.org/about/licence/

IntroductiontoPostgreSQLExtensionNetworkWhenyouwanttopublishyourmodule,youcouldstartwritingpackagingscriptsforeachofthedistributionsystemsforeveryoperatingsystem.ThisisthewaythePostgreSQLextensionshavebeendistributedinthepast.Thatdistributionsystemhasnotbeenveryfriendlytotheopensourcecommunity,orverywellreceived.Inanefforttomakeextensionpublicationmorepalatable,agroupofopensourcewritersandbackingcompaniesgottogetherandfoundedthePostgreSQLExtensionNetwork(PGXN).

ThePostgreSQLExtensionNetworkhttp://pgxn.org/providesacentralrepositoryforyouropensourceextensions.Bythekindnessofthemaintainers,italsoprovidesinstallationscriptsforyourextensionsthatwillworkonmostofthepopularPostgreSQLdeploymentoperatingsystems.

http://pgxn.org/

SigninguptopublishyourextensionTosignuptopublishyourextension,performthefollowingsteps:

1. Startbyrequestinganaccountonthemanagementpage:http://manager.pgxn.org.2. ClickonRequestAccountandfillinyourpersonalinformation.ThePostgreSQL

ExtensionNetworkfolkswillgetbacktoyouviae-mail.Enrollmentrequestsarecurrentlyprocessedbyanactualhuman,sothee-mailresponsewillnotbeimmediate.

3. Clickontheprovidedlinkinthee-mailtoconfirmyouraccountandsetanewpasswordonthePGXNwebsite:

4. Youwillthenbepromptedtocreateapasswordforyouraccount:

http://manager.pgxn.org

5. Setapasswordthatyouwillremember,andconfirmbytypingitagain.ClickonChangeandyouwillbewelcomedtothesite:

Thatisallthereistogettingsignedup.Onceyouhaveyournewaccountsetup,youcandoafewthingsthatwillmakePostgreSQLextensionprogrammingmuchmorepainless.

CreatinganextensionprojecttheeasywayFirst,let’sinstallsomeutilitypackagesthatwillcreatealotofboilerplatefilesthatwehavealreadydescribedinearliersections.ThecommandsbelowareforaDebian/Ubuntusystem:

apt-getinstallruby

apt-getinstallrubygems

apt-getinstallruby1.8-dev

apt-getinstalllibopenssl-ruby1.8

geminstallrubygems-update

/var/lib/gems/1.8/bin/update_rubygems

geminstallpgxn_utils

Youwillnowfindthatyouhaveautilityinstallednamedpgxn-utils.Thisutilitymakesitsupersimpletocreateanextensionproject.

pgxn-utilsskeletonmyextension

createmyextension

createmyextension/myextension.control

createmyextension/META.json

createmyextension/Makefile

createmyextension/README.md

createmyextension/doc/myextension.md

createmyextension/sql/myextension.sql

createmyextension/sql/uninstall_myextension.sql

createmyextension/test/expected/base.out

createmyextension/test/sql/base.sql

Wow!Allofthefilesthatwehavementionedsofarjustgotcreatedinasinglestep.Severalfilesalsogotcreatedtosupporttheoldcontribstyleofdeployment.Thenextfewsectionswillshowwhichonesareimportanttoyouforextensiondevelopment.

Thispackagemanagementsystemhasonenotablerestriction.IncontrasttoPostgreSQL,whichallowsversionnumberstobeanyalphanumerictext,thispackagemanagementrequiresversionnumberstofollowtherulesofsemanticversioning.

Thisversionformatincludesmajorversion,minorversion,andreleasenumberintheformatmajor.minor.release.Thisistoassistthepackagemanagerininstallingyourpackageonmultipleoperatingsystemplatforms.Justgowithit,you’llthankuslater.

ProvidingthemetadataabouttheextensionTherearethreefilesusedtoprovidedataabouttheextension.ThePostgreSQLExtensionNetworkusesoneofthemonthewebsite,META.json,forsearchcriteriaanddescriptiontextfortheextension.META.jsonwillbelocatedinmyextension/META.json.

Hereisanexample:

{

"name":"myextension",

"abstract":"Ashortdescription",

"description":"Alongdescription",

"version":"0.0.1",

"maintainer":"Themaintainer'sname",

"license":"postgresql",

"provides":{

"myextension":{


"file":"sql/myextension.sql",

"docfile":"doc/myextension.md",

"version":"0.0.1"

}

},

"release_status":"unstable",

"generated_by":"Themaintainer'sname",

"meta-spec":{

"version":"1.0.0",

"url":"http://pgxn.org/meta/spec.txt"

}

}

Youshouldaddsomesectionstoittodescribeyourkeywordsandanyadditionalresourcesthatyoumakeavailabletotheuser.Thesesectionswouldlooklikethis:

"tags":[

"curescancer",

"myextension",

"createsworldpeace"

],

"resources":{

"bugtracker":

{"web":"https://github.com/myaccount/myextension/issues/"},

"repository":{

"type":"git",

"url":"git://github.com/myaccount/myextension.git",

"web":"https://github.com/myaccount/myextension/"

}

}

Thecompletefilewouldthenlooklikethis:

{

"name":"myextension",


"description":"Alongdescription",

"version":"0.0.1",

"maintainer":"Themaintainer'sname",

"license":"postgresql",

"provides":{

"myextension":{


"file":"sql/myextension.sql",

"docfile":"doc/myextension.md",

"version":"0.0.1"

}

},

"release_status":"unstable",

"generated_by":"Themaintainer'sname",

"meta-spec":{

"version":"1.0.0",

"url":"http://pgxn.org/meta/spec.txt"

}

"tags":[

"curescancer",

"myextension",

"createsworldpeace"

],

"resources":{

"bugtracker":

{"web":"https://github.com/myaccount/myextension/issues/"},

"repository":{

"type":"git",

"url":"git://github.com/myaccount/myextension.git",

"web":"https://github.com/myaccount/myextension/"

}

}

}

ThenextfilethatyouwillneedtomodifyisREADME.md.Thisfileislocatedinmyextension/README.md.Anexampleisprovidedwiththecodethataccompaniesthisbook.Duetothelength,itwillnotbereproducedhere.Thisfileisdistributedalongwithyourextension.Itisamarkdownsyntaxfilethatismeantforhumanconsumption.Describeanythingyoulikeinit.MineincludesarecipeforDönerKebabs.Quitetasty!Butmostimportantly,putanicelongdescriptionofthebenefitsandeaseofuseofyourextension.Finally,wecometodoc/myextension.md.ThisfileisusedbythePostgreSQLExtensionNetworktoprovideaverynicelandingpageforyourextension.Itwilllooksomethinglikethis:

Thisfileisformattedwithmarkdown.Youmayuseseveraldifferentmarkupsyntaxeshere.Adiscussionofwikimarkupisbeyondthescopeofthisdescription,buttheformattingthatisintheexampleislikelytobeallyouwilleverneedanyway.

Hereisanexampleofthecontentofthefile:

myextension

===========

Synopsis

--------

Showabriefsynopsisoftheextension.

Description

-----------

Alongdescription

Usage

-----

Showusage.

Support

-------

Thereisissuestracker?Github?Putthisinformationhere.

Author

------

[Themaintainer'sname]

CopyrightandLicense

---------------------

Copyright(c)2012Themaintainer'sname.

Filloutthefilewithsomedescriptivenarrativeaboutyourextension.Addanythingthatyouthinkmightberelevanttotheuserthatisevaluatingyourextensionbeforemakingadecisiontoinstallit.ThisisyourchancetoimpressthemassesofPostgreSQLdevelopers.Sodon’tbeshyhere.

WritingyourextensioncodePutyourSQLcodeinthefilethatwasprovidedforyouinmyextension/sql/myextension.sql.Thisfileshouldcontainalloftheobjectsthatmakeupyourextension.

/*myextension.sql*/

—complainifscriptissourcedinpsql,ratherthanviaCREATEEXTENSION

\echoUse"CREATEEXTENSIONmyextension"toloadthisfile.\quit

CREATEFUNCTIONfeed_the_hungry()...

YoucanprovideanyadditionalSQLfilesinthesamedirectoryformaintainingversionsasdescribedintheExtensionversionssection.Anythingnamed*.sqlthatislocatedinthisdirectorywillbeincludedinthedistribution.

CreatingthepackageToultimatelysubmitourextensiontothePostgreSQLExtensionNetwork,weneedtopackageallthefilesintoasingle.zipfile.Assumingwe’refollowinggoodpractices,andwe’rekeepingallofoursourcecodeinahandyGitrepository,wecancreatethepackagethroughasimpleGitcommand.Trythisoneonforsize:

gitarchive--formatzip--prefix=myextension-0.0.1/\

--output~/Desktop/myextension-0.0.1.zipmaster

ThiscommandwillcreateapackageforyouthatissuitableforsubmissiontothePostgreSQLExtensionNetwork.Allweneedtodonowissubmitit.

SubmittingthepackagetoPGXNNowthatyouhaveaniceZIPfileinhand,youcangotothePostgreSQLExtensionNetworkandmakeyouraccomplishmentavailabletothecommunity.

1. Startbygoingtohttp://www.pgxn.org:

2. AtthebottomofthepageisalinknamedReleaseIt.ClickonthelinkandyouwillbetakentothePGXNManagerwhereyoushouldloginwiththeusernameandpasswordthatyoucreatedinthefirstsection:

http://www.pgxn.org

3. ClickonthelinkUploadaDistribution.ThiswillbringyoutothescreenwhereyoucanuploadtheZIPfilethatyoucreatedintheCreatingthepackagesection:

4. BrowseyourcomputerfortheZIPfileanduploadittothePostgreSQLExtensionNetwork.

That’sit.ThanksagainforcontributingtothePostgreSQLcommunity.

InstallinganextensionfromPGXNThePostgreSQLExtensionNetworkprovidesaplatform-independenttooltoinstallPostgreSQLextensions.ThistooliswritteninPython,andusesthePythoninstallationsystemfordistributingitself.ThisishandybecausethePythondistributionsystemexistsvirtuallyoneveryPostgreSQLsupportableplatformandmakesitverysimpletogetPostgreSQLextensionsdistributedtothecommunity.Theextensioninstallerworkswithasinglesetofinstructionsonalltargets:

easy_installpgxnclient

Installingpgxncli.pyscriptto/usr/local/bin

Installingpgxnscriptto/usr/local/bin

Processingdependenciesforpgxnclient

Finishedprocessingdependenciesforpgxnclient

NowyouhavethetoolsinstalledtomanagePostgreSQLextensionsprovidedbythePostgreSQLExtensionNetwork.

Installingextensionsisreallysimple.Forexample,ifwehadarequirementtouseanewtinyintdatatype,wecouldadditwiththiscommand:

pgxninstalltinyint

INFO:bestversion:tinyint0.1.1

INFO:saving/tmp/tmpKvr0kM/tinyint-0.1.1.zip

INFO:unpacking:/tmp/tmpKvr0kM/tinyint-0.1.1.zip

INFO:buildingextension…

Theextensionisnowavailableinthesharedextensionsdirectoryonyourmachine.Toactivateitforanydatabase,youwouldusethecommandthatwestartedthechapterwith:

CREATEEXTENSIONtinyint;

Youwillthenseetheconfirmationtextlettingyouknowthattinyinthasbeenadded:

CREATEEXTENSION

Younowhavetheextensionavailableforuseinyourlocaldatabase.Enjoy!

SummaryWow,thishasbeenalonghardroadtowardgettinganextensionconfiguredandinstalled.Wehaveusedprogrammingskills,systemadministrativeskills,databaseadministrativeskills,andwikiediting.Alongtheway,wesawsomeRuby,Python,shellscripting,PL/pgSQL,andMediaWiki.

Believeitornot,thisisthesimplifiedprocess.Hardtoimagine,eh?Well,continuousworkisbeingdoneonthePostgreSQLExtensionNetworktofurthersimplifythiscatastropheofadevelopmentsystem.MythanksgoouttoDavidE.Wheelerandcrewformakingthisnewsystemavailable.Astheframeworknowexiststohelpwiththetask,therewillbedramaticimprovementscominginthemonthsandyearsahead.

NowthatI’mdonecomplainingaboutit,thisextensionsystemisactuallyrevolutionary.Isaythisbecausenootherdatabaseplatformprovidesanysuchframeworkatall.PostgreSQLleadsthepackwhenitcomestotheabilitytomakechangestothebasicfunctionalityoftheproduct.ThefactthatextensionscanbeinstalledandremovedfromtheproductisanindicatorofhowinvitingPostgreSQListotheopensourcecommunity.

Extendittodowhateveryouwant,andthey’llgiveyouthetoolstodoit.ThismakesaPostgreSQLservertheperfectframeworktouseforyourdataprocessingneeds.

Inthenextchapter,wewilllearnmoreaboutPostgreSQLasanextensibledatabaseandlookathowtocreateuser-defineddatatypesandoperators.

Chapter14.PostgreSQLasanExtensibleRDBMSPostgreSQLisanextensibledatabase.Ihopeyou’velearnedthismuchbynow.Itisextensiblebyvirtueofthedesignthatithas.Asdiscussedbefore,PostgreSQLusesacatalog-drivendesign.Infact,PostgreSQLismorecatalog-driventhanmostofthetraditionalrelationaldatabases.Thekeybenefithereisthatthecatalogscanbechangedoraddedto,inordertomodifyorextendthedatabasefunctionality.PostgreSQLalsosupportsdynamicloading,thatis,auser-writtencodecanbeprovidedasasharedlibrary,andPostgreSQLwillloaditasrequired.

Extensibilityiscriticalformanybusinesses,whichhaveneedsthatarespecifictothatbusinessorindustry.Sometimes,thetoolsprovidedbythetraditionaldatabasesystemsdonotfulfillthoseneeds.Peopleinthosebusinessesknowbesthowtosolvetheirparticularproblems,buttheyarenotexpertsindatabaseinternals.Itisoftennotpossibleforthemtocookuptheirowndatabasekernelormodifythecoreorcustomizeitaccordingtotheirneeds.Atrulyextensibledatabasewillthenallowyoutodothefollowing:

Solvedomain-specificproblemsinaseamlessway,likeanativesolutionBuildcompletefeatureswithoutmodifyingthecoredatabaseengineExtendthedatabasewithoutinterruptingavailability

PostgreSQLnotonlyallowsyoutodoalloftheprecedingthings,butalsodoesthese,andmorewithutmostease.Intermsofextensibility,youcandothefollowingthingsinaPostgreSQLdatabase:

1. Createyourowndatatypes2. Createyourownfunctions3. Createyourownaggregates4. Createyourownoperators5. Createyourownindexaccessmethods(operatorclasses)6. Createyourownserverprogramminglanguage7. Createforeigndatawrappers(SQL/MED)andforeigntables

Sofarinthisbook,youlearnedtocreatefunctionsandtriggersinvariousprogramminglanguagesavailableinPostgreSQL,aswellascreateuser-definedtypes.Youalsolearnedhowtousethesefunctionsintriggersandrules.Asyoucansee,therearemanymoretypesofextensionsyoucandotothedatabase,andthisprovidesyouwithallthetoolsyouneedtocustomizeandextendthedatabasetosuityourbusinessneeds.Beforewediscussthepreviouslymentionedcasesbriefly,let’stakealookatwhatyoucan’textendinPostgreSQL.

Whatcan’tbeextended?AlthoughPostgreSQLisanextensibleplatform,therearecertainthingsthatyoucan’tdoorchangewithoutexplicitlydoingafork,asfollows:

1. Youcan’tchangeorpluginanewstorageengine.IfyouarecomingfromtheMySQLworld,thismightannoyyoualittle.However,PostgreSQL’sstorageengineistightlycoupledwithitsexecutorandtherestofthesystem,whichhasitsownbenefits.

2. Youcan’tpluginyourownplanner/parser.Onecanargueforandagainsttheabilitytodothat,butatthemoment,theplanner,parser,optimizer,andsoonarebakedintothesystemandthereisnopossibilityofreplacingthem.Therehasbeensometalkonthistopic,andifyouareofthecuriouskind,youcanreadsomeofthediscussionathttp://bit.ly/1yRMkK7.

3. WewillnowbrieflydiscusssomemoreoftheextensibilitycapabilitiesofPostgreSQL.Wewillnotdivedeepintothetopics,butwewillpointyoutotheappropriatelinkwheremoreinformationcanbefound.Thechaptermaterialwillserveasaneasy-to-understandintroductorytutorialonthesubjectmatter.Itisbynomeansacomprehensivediscussionofthetopics.

http://bit.ly/1yRMkK7

CreatinganewoperatorNow,let’stakelookathowwecanaddanewoperatorinPostgreSQL.Addingnewoperatorsisnottoodifferentfromaddingnewfunctions.Infact,anoperatorissyntacticallyjustadifferentwaytouseanexistingfunction.Forexample,the+operatorcallsabuilt-infunctioncallednumeric_addandpassesitthetwoarguments.

Whenyoudefineanewoperator,youmustdefinethedatatypesthattheoperatorexpectsasargumentsanddefinewhichfunctionistobecalled.

Let’stakealookathowtodefineasimpleoperator.YouhavetousetheCREATEOPERATORcommandtocreateanoperator.

InChapter2,ServerProgrammingEnvironments,wewroteafunctiontocalculatetheFibonaccinumberofagiveninteger.Let’susethatfunctiontocreateanewFibonaccioperator,##,whichwillhaveanintegeronitsleft-handside:

CREATEOPERATOR##(PROCEDURE=fib,LEFTARG=integer);

Now,youcanusethisoperatorinyourSQLtocalculateaFibonaccinumber:

testdb=#SELECT12##;

?column?

----------

144

(1row)

Notethatwedefinedthattheoperatorwillhaveanintegerontheleft-handside.Ifyoutrytoputavalueontheright-handsideoftheoperator,youwillgetanerror:

postgres=#SELECT##12;

ERROR:operatordoesnotexist:##integeratcharacter8

HINT:Nooperatormatchesthegivennameandargumenttype(s).Youmight


STATEMENT:select##12;

ERROR:operatordoesnotexist:##integer

LINE1:select##12;

^

HINT:Nooperatormatchesthegivennameandargumenttype(s).Youmight


OverloadinganoperatorOperatorscanbeoverloadedinthesamewayasfunctions.Thismeans,thatanoperatorcanhavethesamenameasanexistingoperatorbutwithadifferentsetofargumenttypes.Morethanoneoperatorcanhavethesamename,buttwooperatorscan’tsharethesamenameiftheyacceptthesametypesandpositionsofthearguments.Aslongasthereisafunctionthatacceptsthesamekindandnumberofargumentsthatanoperatordefines,itcanbeoverloaded.

Let’soverridethe##operatorwedefinedinthelastexample,andalsoaddtheabilitytoprovideanintegerontheright-handsideoftheoperator:

CREATEOPERATOR##(PROCEDURE=fib,RIGHTARG=integer);

Now,runningthesameSQL,whichresultedinanerrorlasttime,shouldsucceed,asshownhere:

testdb=#SELECT##12;

?column?

----------

144

(1row)

YoucandroptheoperatorusingtheDROPOPERATORcommand.

NoteYoucanreadmoreaboutcreatingandoverloadingnewoperatorsinthePostgreSQLdocumentationathttp://www.postgresql.org/docs/current/static/sql-createoperator.htmlandhttp://www.postgresql.org/docs/current/static/xoper.html.

Thereareseveraloptionalclausesintheoperatordefinitionthatcanoptimizetheexecutiontimeoftheoperatorsbyprovidinginformationaboutoperatorbehavior.Forexample,youcanspecifythecommutatorandthenegatorofanoperatorthathelptheplannerusetheoperatorsinindexscans.Youcanreadmoreabouttheseoptionalclausesathttp://www.postgresql.org/docs/current/static/xoper-optimization.html.

SincethischapterisjustanintroductiontotheadditionalextensibilitycapabilitiesofPostgreSQL,wewilljustintroduceacoupleofoptimizationoptions;anyseriousproductionqualityoperatordefinitionsshouldincludetheseoptimizationclauses,ifapplicable.

http://www.postgresql.org/docs/current/static/sql-createoperator.html

http://www.postgresql.org/docs/current/static/xoper.html

http://www.postgresql.org/docs/current/static/xoper-optimization.html

OptimizingoperatorsTheoptionalclausestellthePostgreSQLserverabouthowtheoperatorsbehave.Theseoptionscanresultinconsiderablespeedupsintheexecutionofqueriesthatusetheoperator.However,ifyouprovidetheseoptionsincorrectly,itcanresultinaslowdownofthequeries.Let’stakealookattwooptimizationclausescalledcommutatorandnegator.

COMMUTATORThisclausedefinesthecommuteroftheoperator.AnoperatorAisacommutatorofoperatorBifitfulfilsthefollowingcondition:

xAy=yBx.

Itisimportanttoprovidethisinformationfortheoperatorsthatwillbeusedinindexesandjoins.Asanexample,thecommutatorfor>is<,andthecommutatorof=is=itself.

Thishelpstheoptimizertofliptheoperatorinordertouseanindex.Forexample,considerthefollowingquery:

SELECT*FROMemployeeWHEREnew_salary>salary;

Iftheindexisdefinedonthesalarycolumn,thenPostgreSQLcanrewritetheprecedingqueryasshown:

SELECT*fromemployeeWHEREsalary<new_salary

ThisallowsPostgreSQLtousearangescanontheindexcolumnsalary.Forauser-definedoperator,theoptimizercanonlydothisfliparoundifthecommutatorofauser-definedoperatorisdefined:

CREATEOPERATOR>(LEFTARG=integer,RIGHTARG=integer,PROCEDURE=comp,

COMMUTATOR=<)

NEGATORThenegatorclausedefinesthenegatoroftheoperator.Forexample,<>isanegatorof=.Considerthefollowingquery:

SELECT*FROMemployeeWHERENOT(dept=10);

Since<>isdefinedasanegatorof=,theoptimizercansimplifytheprecedingqueryasfollows:

SELECT*FROMemployeeWHEREdept<>10;

YoucanevenverifythatusingtheEXPLAINcommand:

postgres=#EXPLAINSELECT*FROMemployeeWHERENOTdept='WATERMGMNT';

QUERYPLAN

---------------------------------------------------------

ForeignScanonemployee(cost=0.00..1.10rows=1width=160)

Filter:((dept)::text<>'WATERMGMNT'::text)

ForeignFile:/Users/usamadar/testdata.csv

ForeignFileSize:197

(4rows)

CreatingindexaccessmethodsSofarinthisbook,youcameacrossexamplesofcreatingnewdatatypesoruser-definedtypesandoperators.Whatwehaven’tdiscussedsofarishowtoindexthesetypes.InPostgreSQL,anindexismoreofaframeworkthatcanbeextendedorcustomizedforusingdifferentstrategies.Inordertocreatenewindexaccessmethods,wehavetocreateanoperatorclass.Let’stakealookatasimpleexample.

Let’sconsiderascenariowhereyouhavetostoresomespecialdatasuchasanIDorasocialsecuritynumberinthedatabase.Thenumbermaycontainnon-numericcharacters,soitisdefinedasatexttype:

CREATETABLEtest_ssn(ssntext);

INSERTINTOtest_ssnVALUES('222-11-020878');

INSERTINTOtest_ssnVALUES('111-11-020978');

Let’sassumethatthecorrectorderforthisdataissuchthatitshouldbesortedonthelastsixdigitsandnottheASCIIvalueofthestring.

Thefactthatthesenumbersneedauniquesortorderpresentsachallengewhenitcomestoindexingthedata.ThisiswherePostgreSQLoperatorclassesareuseful.Anoperatorallowsausertocreateacustomindexingstrategy.

CreatinganindexingstrategyisaboutcreatingyourownoperatorsandusingthemalongsideanormalB-tree.

Let’sstartbywritingafunctionthatchangestheorderofdigitsinthevalueandalsogetsridofthenon-numericcharactersinthestringtobeabletocomparethembetter:

CREATEORREPLACEFUNCTIONfix_ssn(text)

RETURNStextAS$$

BEGIN

RETURNsubstring($1,8)||replace(substring($1,1,7),'-','');

END;

$$LANGUAGE'plpgsql'IMMUTABLE;

Let’srunthefunctionandverifythatitworks:

testdb=#SELECTfix_ssn(ssn)FROMtest_ssn;

fix_ssn

-------------

02087822211

02097811111

(2rows)

Beforeanindexcanbeusedwithanewstrategy,wemayhavetodefinesomemorefunctionsdependingonthetypeofindex.Inourcase,weareplanningtouseasimpleB-tree,soweneedacomparisonfunction:

CREATEORREPLACEFUNCTIONssn_compareTo(text,text)

RETURNSintAS

$$

BEGIN

IFfix_ssn($1)<fix_ssn($2)

THEN

RETURN-1;

ELSIFfix_ssn($1)>fix_ssn($2)

THEN

RETURN+1;

ELSE

RETURN0;

ENDIF;

END;

$$LANGUAGE'plpgsql'IMMUTABLE;

It’snowtimetocreateouroperatorclass:

CREATEOPERATORCLASSssn_ops

FORTYPEtextUSINGbtree

AS

OPERATOR1<,

OPERATOR2<=,

OPERATOR3=,

OPERATOR4>=,

OPERATOR5>,

FUNCTION1ssn_compareTo(text,text);

Youcanalsooverloadthecomparisonoperatorsifyouneedtocomparethevaluesinaspecialway,andusethefunctionsinthecompareTofunctionaswellasprovidethemintheCREATEOPERATORCLASScommand.

Wewillnowcreateourfirstindexusingourbrandnewoperatorclass:

CREATEINDEXidx_ssnONtest_ssn(ssnssn_ops);

Wecancheckwhethertheoptimizeriswillingtouseourspecialindex,asfollows:

testdb=#SETenable_seqscan=off;

testdb=#EXPLAINSELECT*FROMtest_ssnWHEREssn='02087822211';

QUERYPLAN

------------------------------------------------------------------

IndexOnlyScanusingidx_ssnontest_ssn(cost=0.13..8.14rows=1

width=32)

IndexCond:(ssn='02087822211'::text)

(2rows)

Therefore,wecanconfirmthattheoptimizerisabletouseournewindex.

YoucanreadaboutindexaccessmethodsinthePostgreSQLdocumentationathttp://www.postgresql.org/docs/current/static/xindex.html.

http://www.postgresql.org/docs/current/static/xindex.html

Creatinguser-definedaggregatesUser-definedaggregatefunctionsareprobablyauniquePostgreSQLfeature,yettheyarequiteobscureandperhapsnotmanypeopleknowhowtocreatethem.However,onceyouareabletocreatethisfunction,youwillwonderhowyouhavelivedforsolongwithoutusingthisfeature.

Thisfunctionalitycanbeincrediblyuseful,becauseitallowsyoutoperformcustomaggregatesinsidethedatabase,insteadofqueryingallthedatafromtheclientanddoingacustomaggregateinyourapplicationcode,thatis,thenumberofhitsonyourwebsiteperminutefromaspecificcountry.

PostgreSQLhasaverysimpleprocessfordefiningaggregates.Aggregatescanbedefinedusinganyfunctionsandinanylanguagesthatareinstalledinthedatabase.HerearethebasicstepstobuildinganaggregatefunctioninPostgreSQL:

1. Defineastartfunctionthatwilltakeinthevaluesofaresultset;thisfunctioncanbedefinedinanyPLlanguageyouwant.

2. Defineanendfunctionthatwilldosomethingwiththefinaloutputofthestartfunction.ThiscanbeinanyPLlanguageyouwant.

3. DefinetheaggregateusingtheCREATEAGGREGATEcommand,providingthestartandendfunctionsyoujustcreated.

Let’sstealanexamplefromthePostgreSQLwikiathttp://wiki.postgresql.org/wiki/Aggregate_Median.

Inthisexample,wewillcalculatethestatisticalmedianofasetofdata.Forthispurpose,wewilldefinestartandendaggregatefunctions.

Let’sdefinetheendfunctionfirst,whichtakesanarrayasaparameterandcalculatesthemedian.Weareassumingherethatourstartfunctionwillpassanarraytothefollowingendfunction:

CREATEFUNCTION_final_median(anyarray)RETURNSfloat8AS$$

WITHqAS

(

SELECTval

FROMunnest($1)val

WHEREVALISNOTNULL

ORDERBY1

),

cntAS

(

SELECTCOUNT(*)AScFROMq

)

SELECTAVG(val)::float8

FROM

(

SELECTvalFROMq

LIMIT2-MOD((SELECTcFROMcnt),2)

http://wiki.postgresql.org/wiki/Aggregate_Median

OFFSETGREATEST(CEIL((SELECTcFROMcnt)/2.0)-1,0)

)q2;

$$LANGUAGEsqlIMMUTABLE;

Now,wecreatetheaggregateasshowninthefollowingcode:

CREATEAGGREGATEmedian(anyelement)(

SFUNC=array_append,

STYPE=anyarray,

FINALFUNC=_final_median,

INITCOND='{}'

);

Thearray_appendstartfunctionisalreadydefinedinPostgreSQL.Thisfunctionappendsanelementtotheendofanarray.

Inourexample,thestartfunctiontakesallthecolumnvaluesandcreatesanintermediatearray.Thisarrayispassedontotheendfunction,whichcalculatesthemedian.

Now,let’screateatableandsometestdatatorunourfunction:

testdb=#CREATETABLEmedian_test(tinteger);

CREATETABLE

testdb=#INSERTINTOmedian_testSELECTgenerate_series(1,10);

INSERT010

Thegenerate_seriesfunctionisasetreturningfunctionthatgeneratesaseriesofvalues,fromstarttostopwithastepsizeofone.

Now,weareallsettotestthefunction:

testdb=#SELECTmedian(t)FROMmedian_test;

median

--------

5.5

(1row)

Themechanicsoftheprecedingexamplearequiteeasytounderstand.Whenyouruntheaggregate,thestartfunctionisusedtoappendallthetabledatafromcolumntintoanarrayusingtheappend_arrayPostgreSQLbuilt-in.Thisarrayispassedontothefinalfunction,_final_median,whichcalculatesthemedianofthearrayandreturnstheresultinthesamedatatypeastheinputparameter.Thisprocessisdonetransparentlytotheuserofthefunctionwhosimplyhasaconvenientaggregatefunctionavailabletothem.

Youcanreadmoreabouttheuser-definedaggregatesinthePostgreSQLdocumentationinmuchmoredetailathttp://www.postgresql.org/docs/current/static/xaggr.html.

http://www.postgresql.org/docs/current/static/xaggr.html

UsingforeigndatawrappersPostgreSQLforeigndatawrappers(FDW)areanimplementationofSQLManagementofExternalData(SQL/MED),whichisastandardaddedtoSQLin2013.

FDWsaredriversthatallowPostgreSQLdatabaseuserstoreadandwritedatatootherexternaldatasources,suchasotherrelationaldatabases,NoSQLdatasources,files,JSON,LDAP,andevenTwitter.

YoucanquerytheforeigndatasourcesusingSQLandcreatejoinsacrossdifferentsystemsorevenacrossdifferentdatasources.

Thereareseveraldifferenttypesofdatawrappersdevelopedbydifferentdevelopersandnotallofthemareproductionquality.YoucanseeaselectlistofwrappersonthePostgreSQLwikiathttp://wiki.postgresql.org/wiki/Foreign_data_wrappers.

AnotherlistofFDWscanbefoundonPGXNathttp://pgxn.org/tag/fdw/.

Let’stakelookatasmallexampleofusingfile_fdwtoaccessdatainaCSVfile.

First,youneedtoinstallthefile_fdwextension.IfyoucompiledPostgreSQLfromthesource,youwillneedtoinstallthefile_fdwcontribmodulethatisdistributedwiththesource.Youcandothisbygoingintothecontrib/file_fdwfolderandrunningmakeandmakeinstall.Ifyouusedaninstallerorapackageforyourplatform,thismodulemighthavebeeninstalledautomatically.

Oncethefile_fdwmoduleisinstalled,youwillneedtocreatetheextensioninthedatabase:

postgres=#CREATEEXTENSIONfile_fdw;

CREATEEXTENSION

Let’snowcreateasampleCSVfilethatusesthepipe,|,asaseparatorandcontainssomeemployeedata:

$cattestdata.csv

AARON,ELVIAJ|WATERRATETAKER|WATERMGMNT|81000.00|73862.00

AARON,JEFFERYM|POLICEOFFICER|POLICE|74628.00|74628.00

AARON,KIMBERLEIR|CHIEFCONTRACTEXPEDITER|FLEET

MANAGEMNT|77280.00|70174.00

Now,weshouldcreateaforeignserverthatisprettymuchaformalitybecausethefileisonthesameserver.Aforeignservernormallycontainstheconnectioninformationthataforeigndatawrapperusestoaccessanexternaldataresource.Theserverneedstobeuniquewithinthedatabase:

CREATESERVERfile_serverFOREIGNDATAWRAPPERfile_fdw;

Thenextstep,istocreateaforeigntablethatencapsulatesourCSVfile:

CREATEFOREIGNTABLEemployee(

emp_nameVARCHAR,

job_titleVARCHAR,

http://wiki.postgresql.org/wiki/Foreign_data_wrappers

http://pgxn.org/tag/fdw/

deptVARCHAR,

salaryNUMERIC,

sal_after_taxNUMERIC

)SERVERfile_server

OPTIONS(format'csv',header'false',filename

'/home/pgbook/14/testdata.csv',delimiter'|',null'');'');

TheCREATEFOREIGNTABLEcommandcreatesaforeigntableandthespecificationsofthefileareprovidedintheOPTIONSsectionoftheprecedingcode.Youcanprovidetheformat,andifthefirstlineofthefileisaheader(header'false'),inourcasethereisnofileheader.

Wethenprovidethenameandpathofthefileandthedelimiterusedinthefile,whichinourcaseisthepipesymbol|.Inthisexample,wealsospecifythatthenullvaluesshouldberepresentedasanemptystring.

Let’srunaSQLcommandonourforeigntable:

postgres=#select*fromemployee;

-[RECORD1]-+-------------------------

emp_name|AARON,ELVIAJ

job_title|WATERRATETAKER

dept|WATERMGMNT

salary|81000.00

sal_after_tax|73862.00

-[RECORD2]-+-------------------------

emp_name|AARON,JEFFERYM

job_title|POLICEOFFICER

dept|POLICE

salary|74628.00


-[RECORD3]-+-------------------------

emp_name|AARON,KIMBERLEIR

job_title|CHIEFCONTRACTEXPEDITER

dept|FLEETMANAGEMNT

salary|77280.00


Great,lookslikeourdataissuccessfullyloadedfromthefile.

Youcanalsousethe\dmetacommandtoseethestructureoftheemployeetable:

postgres=#\demployee;

Foreigntable"public.employee"

Column|Type|Modifiers|FDWOptions

---------------+-------------------+-----------+-------------

emp_name|charactervarying||

job_title|charactervarying||

dept|charactervarying||

salary|numeric||

sal_after_tax|numeric||

Server:file_server

FDWOptions:(format'csv',header'false',

filename'/home/pg_book/14/testdata.csv',delimiter'|',"null"'')

Youcanrunexplainonthequerytounderstandwhatisgoingonwhenyourunaqueryontheforeigntable:

postgres=#EXPLAINSELECT*FROMemployeeWHEREsalary>5000;

QUERYPLAN

---------------------------------------------------------

ForeignScanonemployee(cost=0.00..1.10rows=1width=160)

Filter:(salary>5000::numeric)

ForeignFile:/home/pgbook/14/testdata.csv

ForeignFileSize:197

(4rows)

TheALTERFOREIGNTABLEcommandcanbeusedtomodifytheoptions.

NoteMoreinformationaboutthefile_fdwisavailableathttp://www.postgresql.org/docs/current/static/file-fdw.html.

YoucantakealookattheCREATESERVERandCREATEFOREIGNTABLEcommandsinthePostgreSQLdocumentationformoreinformationonthemanyoptionsavailable.Eachoftheforeigndatawrapperscomeswithitsowndocumentationabouthowtousethewrapper.Makesurethatanextensionisstableenoughbeforeitisusedinproduction.ThePostgreSQLcoredevelopmentgroupdoesnotsupportmostoftheFDWextensions.

Ifyouwanttocreateyourowndatawrappers,youcanfindthedocumentationathttp://www.postgresql.org/docs/current/static/fdwhandler.htmlasanexcellentstartingpoint.Thebestwaytolearn,however,istoreadthecodeofotheravailableextensions.

http://www.postgresql.org/docs/current/static/file-fdw.html

http://www.postgresql.org/docs/current/static/fdwhandler.html

SummaryPostgreSQLcanbeextendedinmanymorewaysthanwhatwehavediscussedsofarinthebook.Thisincludestheabilitytoaddnewoperators,newindexaccessmethods,andcreateyourownaggregates.Youcanaccessforeigndatasources,suchasotherdatabases,files,andwebservicesusingPostgreSQLforeigndatawrappers.Thesewrappersareprovidedasextensionsandshouldbeusedwithcaution,asmostofthemarenotofficiallysupported.

EventhoughPostgreSQLisveryextensible,youcan’tpluginanewstorageengineorchangetheparser/plannerandexecutorinterfaces.Thesecomponentsareverytightlycoupledwitheachotherandare,therefore,highlyoptimizedandmature.

IndexA

acquisitioncost/Costofacquisition

add(int,int)functionality,adding/Addingfunctionalitytoadd(int,int)NULLarguments,handling/SmarthandlingofNULLargumentsanynumberofarguments,workingwith/Workingwithanynumberofarguments

add_func.c/add_func.cadd_func.sql.in/add_func.sql.inAFTERtrigger/DisallowingDELETEALTEREXTENSIONADDcommand

URL/WhentocreateanextensionANYparameter/Otherparametersapplicationdesign

about/Applicationdesigndatabases,drawbacks/Databasesareconsideredharmfuldatabases/Databasesareconsideredharmfulencapsulation/EncapsulationPostgreSQL/WhatdoesPostgreSQLoffer?datalocality/Datalocality

argumentsabout/Workingwithanynumberofargumentsrecords,handlingas/Handlingrecordsasargumentsorreturnedvalues

argumenttuplefields,extractingfrom/Extractingfieldsfromanargumenttuple

arrayslooping/LoopingThroughArrays

assert,PL/Pythonusing/Usingassert

audittrailcreating/Creatinganaudittrail

audittrigger/Theaudittrigger

Bbackends

synchronizingbetween/SynchronizingbetweenbackendsBEFOREtrigger/DisallowingDELETEBIRT/Third-partytools

C.controlfile,extension

about/The.controlfileC

additionalresources/AdditionalresourcesforCC++

functions,writingin/WritingfunctionsinC++caching

about/Cachingcanceltrigger/DisallowingDELETECcode,writing

about/BasicguidelinesforwritingCcodememory,allocating/Memoryallocationpalloc(),using/Usepalloc()andpfree()pfree(),using/Usepalloc()andpfree()structures,zerofilling/Zero-fillthestructuresfiles,including/Includefilessymbolnames,public/Publicsymbolnames

Cfunctionabout/ThesimplestCfunction–return(a+b)return(a+b)/ThesimplestCfunction–return(a+b)add_func.c/add_func.cMakefilefunction/MakefileCREATEFUNCTIONadd(int,int)/CREATEFUNCTIONadd(int,int)add_func.sql.infunction/add_func.sql.inwriting/SummaryforwritingaCfunction

Cfunctionserror,reporting/ErrorreportingfromCfunctionserror,states/“Error”statesthatarenoterrorsmessages,senttoclient/Whenaremessagessenttotheclient?

changesauditing/Auditingchanges

CLUSTERstatement/CONNECT,CLUSTER,andRUNONcode

examples/Aboutthisbook’scodeexamplescommit/Doingsomethingatcommit/rollbackCommonLanguageRuntime(CLR)/Procedurallanguagescommunity

about/CommunityCOMMUTATORclause/COMMUTATORcomposite-typearguments,PL/Tcl

passing/Passingcomposite-typeargumentsconditionalexpressions

about/ConditionalexpressionsURL/Conditionalexpressionsloops,withcounters/Loopswithcountersqueryresults,looping/LoopingthroughqueryresultsPERFORMcommandversusSELECTcommand/PERFORMversusSELECTlooping,througharrays/LoopingThroughArrays

conditionaltriggers/ConditionaltriggersCONNECTstatement/CONNECT,CLUSTER,andRUNONcontextmanager

URL/Handlingexceptionscontrib

URL/AdditionalresourcesforCCoordinatedUniversalTime(UTC)/Theaudittriggercost

about/MorecontrolCREATEFUNCTIONadd(int,int)/CREATEFUNCTIONadd(int,int)ctags

URL/AdditionalresourcesforCcursors

returning/Returningcursorsreturnedfromanotherfunction,iteratingover/Iteratingovercursorsreturnedfromanotherfunctionpros/Wrappingupoffunctionsreturningcursorscons/WrappingupoffunctionsreturningcursorsURL/Wrappingupoffunctionsreturningcursors

Ddata

cleaning/Datacleaningpartitioning,acrossmultipleservers/Datapartitioningacrossmultipleserverssplitting/Splittingthedatadistributing/Thedistributionofdatamoving,fromsingletopartitioneddatabase/Movingdatafromthesingletothepartitioneddatabase

databasechanges,fastcapturing/Fastcapturingofdatabasechanges

database,scalingsingle-serverchat,creating/Creatingasimplesingle-serverchattables,splittingovermultipledatabases/Dealingwithsuccess–splittingtablesovermultipledatabasesdata,movingfromsingletopartitioneddatabase/Movingdatafromthesingletothepartitioneddatabase

database-backedsystems,growingways/Whatexpansionplansworkandwhen?biggerserver,movingto/MovingtoabiggerserverMaster-slavereplication/Master-slavereplication–movingreadstoslaveMulti-masterreplication/Multimasterreplication

databaseabstractionlayer/Databasesareconsideredharmfuldatabases,PL/Tcl

accessing/Accessingdatabasesdatachanges

visibility/Visibilityofdatachangesdatacomparisons

operatorsused/Datacomparisonsusingoperatorsdatadefinitionlanguage(DDL)/AuditingchangesDataManipulationLanguage(DML)operation/Workingonasimple“Hey,I’mcalled”triggerdatawrappers

URL/UsingforeigndatawrappersDatum/Interlude–whatisDatum?DBAPI2/Runningqueriesinthedatabaseddl_command_endevent/Creatingeventtriggersddl_command_startevent/Creatingeventtriggersdebugging

manualdebugging,withRAISENOTICE/ManualdebuggingwithRAISENOTICEvisualdebugging/Visualdebugging

debugging,manualexceptions,throwing/Throwingexceptions

URL/Throwingexceptionsfile,loggingto/LoggingtoafileRAISENOTICE,advantages/TheadvantagesofRAISENOTICERAISENOTICE,disadvantages/ThedisadvantagesofRAISENOTICE

DELETEtriggerdisallowing/DisallowingDELETE

developersavailability/Availabilityofdevelopers

don’trepeatyourself(DIY)/DRY–don’trepeatyourselfdynamiclinklibrary(DLL)/Procedurallanguages

EEnterpriseDB

URL/Installingthedebuggerfromthesourceerror

reporting,fromCfunctions/ErrorreportingfromCfunctionsstates/“Error”statesthatarenoterrorsNOTICE/“Error”statesthatarenoterrorsINFO/“Error”statesthatarenoterrorsLOG/“Error”statesthatarenoterrorsreporting,URL/Whenaremessagessenttotheclient?

errorhandling/Generalerrorreportinganderrorhandlingerrorreporting/GeneralerrorreportinganderrorhandlingERRORtrigger/DisallowingDELETEeventtriggers

usecases/Usecasesforcreatingeventtriggerscreating/Creatingeventtriggersddl_command_startevent/Creatingeventtriggersddl_command_endevent/Creatingeventtriggerssql_dropevent/CreatingeventtriggersURL/Creatingeventtriggers,Aroadmapofeventtriggersaudittrail,creating/Creatinganaudittrailroadmap/Aroadmapofeventtriggers

eventtriggers,PL/pgSQLfunctionsTG_TAG/CreatingeventtriggersTG_EVENT/Creatingeventtriggers

exceptions,PL/Pythonhandling/Handlingexceptions

exceptions,RAISENOTICEthrowing/Throwingexceptions

expandeddisplayswitchingto/Switchingtotheexpandeddisplay

extensibility/Whatcan’tbeextended?extension

creating/WhentocreateanextensionURL/Whentocreateanextension,The.controlfileunpackaged/Unpackagedextensionsversions/Extensionversions.controlfile/The.controlfilebuilding/Buildinganextensioninstalling/Installinganextensionviewing/Viewingextensionspublishing/Publishingyourextensioninstalling,fromPGXN/InstallinganextensionfromPGXN

extension,publishingPostgreSQLExtensionNetwork/IntroductiontoPostgreSQLExtensionNetworksigningup/Signinguptopublishyourextensionextensionproject,creating/Creatinganextensionprojecttheeasywaymetadata,providing/Providingthemetadataabouttheextensionextensioncode,writing/Writingyourextensioncodepackage,creating/Creatingthepackage

extensions/add_func.sql.in

Ffile_fdw

URL/Usingforeigndatawrappersfillfactor

URL/Creatingasimplesingle-serverchatforeigndatawrappers(FDW)

using/Usingforeigndatawrappersfunction

used,forconfiguringPL/Proxycluster/ConfiguringthePL/Proxyclusterusingfunctions

functionoverloading/User-definedfunctions

GGeneralInvertedIndex(GIN)/TypeextensibilityGitrepository

URL/Installingthedebuggerfromthesource

Iimmutablefieldstrigger/Theimmutablefieldstriggerindexaccessmethods

creating/CreatingindexaccessmethodsURL/Creatingindexaccessmethods

integersetreturning/Returningasetofintegers

Kkeepitsimplestupid(KISS)/KISS–keepitsimplestupidknearestneighbor(KNN)/Typeextensibility

Llicensing

about/Licensinglight-weightlocks(LWLocks)/Synchronizingbetweenbackendslogtrigger/Alogtriggerloopingsyntax

URL/Loopswithcountersloops

withcounters/Loopswithcountersstatement,terminating/Statementtermination

MMakefilefunction/MakefileMaster-slavereplication/Master-slavereplication–movingreadstoslavemetadata

providing,forextension/ProvidingthemetadataabouttheextensionMulti-masterreplication

about/MultimasterreplicationMultiversionConcurrencyControl(MVCC)/Visibility

NNEGATORclause/NEGATORNEWrecord

modifying/ModifyingtheNEWrecordNULLarguments

handling/SmarthandlingofNULLarguments

Ooperator

newoperator,creating/Creatinganewoperatoroverloading/Overloadinganoperatoroptimizing/Optimizingoperators

operator,optimizingCOMMUTATOR/COMMUTATORNEGATOR/NEGATOR

operatorsused,fordatacomparisons/DatacomparisonsusingoperatorsPostgreSQLdocumentation,URL/Overloadinganoperator

optionalclausesURL/Overloadinganoperator

os.walk()URL/Listingdirectorycontents

OUTparametersandrecords/OUTparametersandrecordsabout/OUTparametersrecords,returning/ReturningrecordsRETURNSTABLE,using/UsingRETURNSTABLEnopredefinedstructure,returningwith/ReturningwithnopredefinedstructureSETOFANY,returning/ReturningSETOFANYvariadicargumentlists/Variadicargumentlists

Ppackage

creating/Creatingthepackagesubmitting,toPGXN/SubmittingthepackagetoPGXN

palloc()using/Usepalloc()andpfree()

parametersabout/Otherparameters

Pentahodataintegration(kettle)/Third-partytoolsPentahoReportServer/Third-partytoolsPERFORMcommand

versusSELECTcommand/PERFORMversusSELECTpfree()

using/Usepalloc()andpfree()pgAdmin3/Third-partytools

installing/InstallingpgAdmin3pgfoundry

URL/AdditionalresourcesforCPGXN

package,submitting/SubmittingthepackagetoPGXNURL/SubmittingthepackagetoPGXNextension,installingfrom/InstallinganextensionfromPGXNFDWsURL/Usingforeigndatawrappers

php5-postgresql/Third-partytoolspl/lolcode

URL/SummaryPL/Perl

using/WhentousePL/Perlinstalling/InstallingPL/Perlfunction/AsimplePL/Perlfunctionfunction,URL/AsimplePL/Perlfunction,Passingandreturningnon-scalartypes,WritingPL/Perltriggersnon-scalartypes,passing/Passingandreturningnon-scalartypesnon-scalartypes,returning/Passingandreturningnon-scalartypestriggers,writing/WritingPL/PerltriggersuntrustedPerl/UntrustedPerl

PL/pgSQLused,forintegritychecks/UsingPL/pgSQLforintegritychecksabout/WhyPL/pgSQL?disadvantages/WhyPL/pgSQL?URL/WhyPL/pgSQL?advantages/WhyPL/pgSQL?

PL/pgSQLdebugger

about/VisualdebuggingURL/Visualdebugginginstalling/InstallingthedebuggerPostgreSQLWindowsinstaller,URL/Installingthedebuggerinstalling,fromsource/InstallingthedebuggerfromthesourcepgAdmin3,installing/InstallingpgAdmin3using/Usingthedebuggeradvantages/Theadvantagesofthedebuggerdisadvantages/Thedisadvantagesofthedebugger

PL/pgSQLfunctionstructure/ThestructureofaPL/pgSQLfunctionarguments,accessing/Accessingfunctionargumentsresults/Actingonthefunction’sresults

PL/pgSQLTRIGGERfunctionOLD,NEW/VariablespassedtothePL/pgSQLTRIGGERfunctionTG_NAME/VariablespassedtothePL/pgSQLTRIGGERfunctionTG_WHEN/VariablespassedtothePL/pgSQLTRIGGERfunctionTG_LEVEL/VariablespassedtothePL/pgSQLTRIGGERfunctionTG_OP/VariablespassedtothePL/pgSQLTRIGGERfunctionTG_RELID/VariablespassedtothePL/pgSQLTRIGGERfunctionTG_TABLE_NAME/VariablespassedtothePL/pgSQLTRIGGERfunctionTG_TABLE_SCHEMA/VariablespassedtothePL/pgSQLTRIGGERfunctionTG_NARGS,TG_ARGV[]/VariablespassedtothePL/pgSQLTRIGGERfunctionTG_TAG/VariablespassedtothePL/pgSQLTRIGGERfunction

PL/Proxyabout/PL/Proxy–thepartitioninglanguageinstalling/InstallingPL/ProxyURL/InstallingPL/Proxysyntax/ThePL/ProxylanguagesyntaxCONNECTstatement/CONNECT,CLUSTER,andRUNONCLUSTERstatement/CONNECT,CLUSTER,andRUNONRUNONstatement/CONNECT,CLUSTER,andRUNONSELECTstatement/SELECTandTARGETTARGETstatement/SELECTandTARGETSPLITstatement/SPLIT–distributingarrayelementsoverseveralpartitionsdata,distributing/Thedistributionofdataconnectionpooling/ConnectionPooling

PL/Proxyclusterconfiguring,functionsused/ConfiguringthePL/Proxyclusterusingfunctionsconfiguring,SQL/MEDused/ConfiguringthePL/ProxyclusterusingSQL/MED

PL/Pythonabout/WhyPL/Python?,QuickintroductiontoPL/Python

function/AminimalPL/Pythonfunctiondatatypeconversions/Datatypeconversionsdocumentation,URL/Datatypeconversionssimplefunctions,writing/WritingsimplefunctionsinPL/Python,Asimplefunctionrecord,returningfromPythonfunction/Functionsreturningarecordtablefunctions/Tablefunctionsqueries,runningindatabase/Runningqueriesinthedatabasetriggerfunctions,writing/WritingtriggerfunctionsinPL/Pythonexceptions,handling/Handlingexceptionsatomicity/AtomicityinPythondebugging/DebuggingPL/Python

PL/Python,debuggingabout/DebuggingPL/Pythonfunctionprogresstracking,plpy.notice()used/Usingplpy.notice()totrackthefunction’sprogressassertused/Usingassertsys.stdout,redirecting/Redirectingsys.stdoutandsys.stderrsys.stderr,redirecting/Redirectingsys.stdoutandsys.stderr

PL/Tclinstalling/InstallingPL/Tclfunction/AsimplePL/TclfunctionStrictfunctions,nullcheckingwith/NullcheckingwithStrictfunctionsparameters/Theparameterformatarrays,passing/Passingandreturningarraysarrays,returning/Passingandreturningarrayscomposite-typearguments,passing/Passingcomposite-typeargumentsdatabase,accessing/Accessingdatabasesfunction,URL/Accessingdatabases,WritingPL/Tcltriggerstriggers,writing/WritingPL/TcltriggersuntrustedTcl/UntrustedTcl

PlpgGetNames()function/Returningarecordplpy.notice()

used,fortrackingfunctionsprogress/Usingplpy.notice()totrackthefunction’sprogress

plugins/Procedurallanguagespolymorphictypes/OtherparametersPostgreSQL

acquisitioncost/Costofacquisitiondevelopers,availability/Availabilityofdeveloperslicensing/Licensingpredictability/Predictabilitycommunity/Communityprocedurallanguages/Procedurallanguages

controls/MorecontrolURL/Morecontroldocumentation,URL/Returningcursorsmanual,URL/AdditionalresourcesforCinternals,URL/AdditionalresourcesforC

postgresql.conffileURL/AdditionalresourcesforC

PostgreSQLdocumentationURL/WritingfunctionsinC++

PostgreSQLExtensionNetworkabout/IntroductiontoPostgreSQLExtensionNetwork

PostgreSQLfunctionscalling/RunningqueriesandcallingPostgreSQLfunctions

PostgreSQLlicenseURL/Publishingyourextension

PostgreSQLVersion9.3functionsURL/MoreinfoonSPI_*functions

predictabilityabout/Predictability

procedurallanguagesabout/Procedurallanguagesthird-partytools/Third-partytoolsplatformcompatibility/Platformcompatibilityapplicationdesign/Applicationdesign

pseudotypesURL/Otherparameters

psycopg2/Third-partytoolsPythonDatabaseAPISpecificationv2.0/RunningqueriesinthedatabasePythonDatabaseAPISpecificationv2.0(DBAPI2)/RunningqueriesinthedatabasePythonImagingLibrary(PIL)module/Generatingthumbnailswhensavingimages

QQCubed/Third-partytoolsqueries,PL/Python

running,indatabase/Runningqueriesinthedatabasesimplequeries,running/Runningsimplequeriespreparedqueries,using/Usingpreparedqueriespreparedqueries,caching/Cachingpreparedqueriesconstructing/Constructingqueries

queryresultslooping/Loopingthroughqueryresults

RRAISENOTICE

manualdebuggingwith/ManualdebuggingwithRAISENOTICEadvantages/TheadvantagesofRAISENOTICEURL/TheadvantagesofRAISENOTICEdisadvantages/ThedisadvantagesofRAISENOTICE

ReadCommitted/Transactionsrecord

returning/Returningarecordreturning,fromPythonfunction/Functionsreturningarecord

recordsandOUTparameters/OUTparametersandrecordsreturning/Returningrecordshandling,asarguments/Handlingrecordsasargumentsorreturnedvaluescomplextypesingletuple,returning/Returningasingletupleofacomplextypefields,extractingfromargumenttype/Extractingfieldsfromanargumenttuplereturntuple,constructing/ConstructingareturntupleDatum/Interlude–whatisDatum?set,returning/Returningasetofrecords

replicationMaster-slavereplication/Master-slavereplication–movingreadstoslaveMulti-masterreplication/Multimasterreplication

return(a+b)/ThesimplestCfunction–return(a+b)RETURNSETOFvariants

about/AsummaryoftheRETURNSETOFvariantsRETURNSTABLE

using/UsingRETURNSTABLEreturntuple

constructing/Constructingareturntuplerollback/Doingsomethingatcommit/rollbackrows

returning,fromfunction/Usingasetreturningfunctionrowsets/SetsandarraysRUNONstatement/CONNECT,CLUSTER,andRUNON

Sschemachanges

preventing/PreventingschemachangesSELECTcommand

versusPERFORMcommand/PERFORMversusSELECTSELECTstatement/SELECTandTARGETserver

data,partitioningacrossmultipleservers/Datapartitioningacrossmultipleservers

serverprogrammingabout/Movingbeyondsimplefunctionsbestpractices/Programmingbestpracticeskeepitsimplestupid(KISS)/KISS–keepitsimplestupiddon’trepeatyourself(DRY)/DRY–don’trepeatyourselfyouain’tgonnaneedit(YAGNI)/YAGNI–youain’tgonnaneeditservice-orientedarchitecture(SOA)/SOA–service-orientedarchitecture

service-orientedarchitecture(SOA)/SOA–service-orientedarchitectureset-returningfunction(tablefunction)

using/Usingasetreturningfunctionset-returningfunctions(SRF)/ReturningarecordSETOFANY

returning/ReturningSETOFANYsets

about/Setsandarraysreturning/Returningsetsintegersets,returning/Returningasetofintegersset-returningfunction,using/Usingasetreturningfunctionrows,returningfromfunction/Usingasetreturningfunction

severprogrammingabout/Whyprogramintheserver?advantages/Wrappingup–whyprogramintheserver?,Easeofmaintenance

single-serverchatspecifications/Creatingasimplesingle-serverchatimplementing/Creatingasimplesingle-serverchat

skytoolsURL/Fastcapturingofdatabasechanges

smtplibURL/Sendingane-mail

sortorderscustom/Customsortorders

SPIused,forsampleCfunction/AsampleCfunctionusingSPI

SPI_*functions/MoreinfoonSPI_*functions

SPI_exec()function/AsampleCfunctionusingSPISPLITstatement

about/SPLIT–distributingarrayelementsoverseveralpartitionsSQL/MED

URL/Thedistributionofdataused,forconfiguringPL/Proxycluster/ConfiguringthePL/ProxyclusterusingSQL/MED

/UsingforeigndatawrappersSQLdatabaseserver

about/Thinkingoutofthe“SQLdatabaseserver”boxthumbnails,creating/Generatingthumbnailswhensavingimagese-mail,sending/Sendingane-maildirectorycontents,listing/Listingdirectorycontents

SQLqueriesrunning,insidedatabase/RunningqueriesandcallingPostgreSQLfunctionssampleCfunction,SPIused/AsampleCfunctionusingSPIdatachanges,visibility/VisibilityofdatachangesSPI_*functions/MoreinfoonSPI_*functions

sql_dropevent/CreatingeventtriggersStrictfunctions

nullcheckingwith/NullcheckingwithStrictfunctionsstructureddata

about/Otherwaystoworkwithstructureddatadatatypes,complex/Complexdatatypesforthemodernworld–XMLandJSONXMLdatatype/XMLdatatypeandreturningdataasXMLfromfunctionsdata,returningasXMLfromfunctions/XMLdatatypeandreturningdataasXMLfromfunctionsdata,returninginJSONformat/ReturningdataintheJSONformat

sys.stderr,PL/Pythonredirecting/Redirectingsys.stdoutandsys.stderr

sys.stdout,PL/Pythonredirecting/Redirectingsys.stdoutandsys.stderr

Ttablefunctions/Tablefunctionstables

splitting,overmultipledatabases/Dealingwithsuccess–splittingtablesovermultipledatabases

Talend/Third-partytoolsTARGETstatement/SELECTandTARGETTcl

URL/Passingcomposite-typeargumentstransactions

about/Transactionsisolationmethods,URL/Transactions

triggerfunction,creating/Creatingthetriggerfunctioncreating/Creatingthetriggersimpletrigger,creating/Workingonasimple“Hey,I’mcalled”triggerauditing/TheaudittriggerDELETEtrigger,disallowing/DisallowingDELETEcanceltrigger/DisallowingDELETEBEFOREtrigger/DisallowingDELETEAFTERtrigger/DisallowingDELETEERRORtrigger/DisallowingDELETETRUNCATEtrigger,disallowing/DisallowingTRUNCATENEWrecord,modifying/ModifyingtheNEWrecordtimestamping/Thetimestampingtriggerimmutablefieldstrigger/Theimmutablefieldstriggerfire,controlling/Controllingwhenatriggeriscalledconditionaltriggers/Conditionaltriggersonspecificfieldchanges/Triggersonspecificfieldchangesfunction,visibility/Visibilityrules,forusing/Mostimportantly–usetriggerscautiously!PL/pgSQLTRIGGERfunction/VariablespassedtothePL/pgSQLTRIGGERfunction

triggerfunctions,PL/Pythonwriting/WritingtriggerfunctionsinPL/Pythoninputs,exploring/Exploringtheinputsofatriggerlogtrigger/Alogtrigger

triggersused,formanagingrelateddata/Managingrelateddatawithtriggers

triggers,PL/Perlwriting/WritingPL/Perltriggers

triggers,PL/Tclwriting/WritingPL/Tcltriggers

OKvalue/WritingPL/TcltriggersSKIPvalue/WritingPL/TcltriggersLISTvalue/WritingPL/TcltriggersOLDvalue/WritingPL/TcltriggersnEWvalue/WritingPL/Tcltriggers$TG_name/WritingPL/Tcltriggers$TG_level/WritingPL/Tcltriggers$TG_when/WritingPL/Tcltriggers$TG_op/WritingPL/Tcltriggers$TG_table_name/WritingPL/Tcltriggers$OLD(i)/WritingPL/Tcltriggers$NEW(i)/WritingPL/Tcltriggers

TRUNCATEtriggerdisallowing/DisallowingTRUNCATE

typeextensibility/Typeextensibility

Uuntrustedlanguages

about/Areuntrustedlanguagesinferiortotrustedones?,Canyouuseuntrustedlanguagesforimportantfunctions?,Willuntrustedlanguagescorruptthedatabase?features/Whyuntrusted?

untrustedPerl/UntrustedPerluntrustedTcl/UntrustedTcluser-definedaggregates

creating/Creatinguser-definedaggregatesexample,URL/Creatinguser-definedaggregatesURL/Creatinguser-definedaggregates

User-definedfunctions(UDF)about/User-definedfunctions

Vvariableparameters

URL/User-definedfunctionsvariables

URL/VariablespassedtothePL/pgSQLTRIGGERfunctionvariadicargumentlists/Variadicargumentlistsversion0callconventions

about/Version0callconventionsURL/Version0callconventions

viewsfunctionsbased/Functionsbasedonviews

visibilityURL/Visibility

visibilityrulesURL/Visibilityofdatachanges

VOLATILEfunction/Visibility

Wwrappers

URL/Usingforeigndatawrappers

XXMLdatatype

about/XMLdatatypeandreturningdataasXMLfromfunctionsURL/XMLdatatypeandreturningdataasXMLfromfunctions

YYii/Third-partytoolsyouain’tgonnaneedit(YAGNI)/YAGNI–youain’tgonnaneedit

Documents

PostgreSQL Server Programming - Second Editionindex-of.co.uk/Database/SQL Database/PostgreSQL... · PostgreSQL Server Programming Second Edition Credits About the Authors About the