Pivotal Greenplum DatabaseContents Release Notes 3 Contents Chapter 3: Pivotal Greenplum 6.1 Release Notes.....14 Release 6.1.1..... 15 Resolved

PRODUCT DOCUMENTATION

Pivotal™ GreenplumDatabase®Version 6.1

Pivotal Greenplum DatabaseDocumentationRev: A03

© 2019 Pivotal Software, Inc.

Copyright Release Notes

2

Notice

Copyright

Privacy Policy | Terms of Use

Copyright © 2019 Pivotal Software, Inc. All rights reserved.

Pivotal Software, Inc. believes the information in this publication is accurate as of its publication date. Theinformation is subject to change without notice. THE INFORMATION IN THIS PUBLICATION IS PROVIDED"AS IS." PIVOTAL SOFTWARE, INC. ("Pivotal") MAKES NO REPRESENTATIONS OR WARRANTIES OF ANYKIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMSIMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Use, copying, and distribution of any Pivotal software described in this publication requires an applicablesoftware license.

All trademarks used herein are the property of Pivotal or their respective owners.

Revised November 2019 (6.1.1)

http://pivotal.io/privacy-policyhttp://pivotal.io/terms-of-use

Contents Release Notes

3

Contents

Chapter 3: Pivotal Greenplum 6.1 Release Notes.................................. 14Release 6.1.1.................................................................................................................................... 15

Resolved Issues..................................................................................................................... 15Release 6.1.0.................................................................................................................................... 16

New Features......................................................................................................................... 16Resolved Issues..................................................................................................................... 17

Upgrading to Greenplum 6.1.............................................................................................................20Deprecated Features.........................................................................................................................21Migrating Data to Greenplum 6........................................................................................................ 22Known Issues and Limitations.......................................................................................................... 23

Chapter 4: Installing and Upgrading Greenplum................................... 24Platform Requirements......................................................................................................................25

Operating Systems................................................................................................................. 25Hardware and Network...........................................................................................................27Storage....................................................................................................................................27Tools and Extensions Compatibility....................................................................................... 28Hadoop Distributions.............................................................................................................. 30

Introduction to Greenplum.................................................................................................................31The Greenplum Master...........................................................................................................32The Segments........................................................................................................................ 32The Interconnect.....................................................................................................................36ETL Hosts for Data Loading.................................................................................................. 38Greenplum Performance Monitoring...................................................................................... 39

Estimating Storage Capacity............................................................................................................. 41Calculating Usable Disk Capacity.......................................................................................... 41Calculating User Data Size.................................................................................................... 42Calculating Space Requirements for Metadata and Logs......................................................42

Configuring Your Systems.................................................................................................................43Disabling SELinux and Firewall Software.............................................................................. 43Recommended OS Parameters Settings............................................................................... 44Synchronizing System Clocks................................................................................................ 51Creating the Greenplum Administrative User.........................................................................51Next Steps.............................................................................................................................. 53

Installing the Greenplum Database Software................................................................................... 54Installing Greenplum Database.............................................................................................. 54Enabling Passwordless SSH.................................................................................................. 55Confirming Your Installation................................................................................................... 56About Your Greenplum Database Installation........................................................................56Next Steps.............................................................................................................................. 56

Creating the Data Storage Areas......................................................................................................57Creating Data Storage Areas on the Master and Standby Master Hosts...............................57Creating Data Storage Areas on Segment Hosts.................................................................. 57Next Steps.............................................................................................................................. 58

Validating Your Systems................................................................................................................... 59Validating Network Performance............................................................................................ 59Validating Disk I/O and Memory Bandwidth...........................................................................60

Initializing a Greenplum Database System.......................................................................................61


4

Overview................................................................................................................................. 61Initializing Greenplum Database.............................................................................................61Setting Greenplum Environment Variables............................................................................ 64Next Steps.............................................................................................................................. 65

Installing Optional Extensions........................................................................................................... 67Procedural Language, Machine Learning, and Geospatial Extensions..................................67Python Data Science Module Package..................................................................................67R Data Science Library Package........................................................................................... 71Greenplum Platform Extension Framework (PXF)................................................................. 74

Installing Additional Supplied Modules..............................................................................................75Configuring Timezone and Localization Settings..............................................................................76

Configuring the Timezone...................................................................................................... 76About Locale Support in Greenplum Database..................................................................... 76Character Set Support............................................................................................................78Setting the Character Set.......................................................................................................80Character Set Conversion Between Server and Client..........................................................81

Upgrading from an Earlier Greenplum 6 Release............................................................................ 84Upgrading from 6.x to a Newer 6.x Release......................................................................... 84Troubleshooting a Failed Upgrade.........................................................................................86

Migrating Data from Greenplum 4.3 or 5..........................................................................................87Preparing the Greenplum 6 Cluster....................................................................................... 87Preparing Greenplum 4.3 and 5 Databases for Backup........................................................ 88Backing Up and Restoring a Database..................................................................................90Completing the Migration........................................................................................................91

Enabling iptables (Optional).............................................................................................................. 93Example iptables Rules..........................................................................................................93

Installation Management Utilities.......................................................................................................96Greenplum Environment Variables................................................................................................... 97

Required Environment Variables............................................................................................97Optional Environment Variables............................................................................................. 97

Example Ansible Playbook................................................................................................................99

Chapter 6: Greenplum Database Administrator Guide........................ 101Greenplum Database Concepts...................................................................................................... 102

About the Greenplum Architecture....................................................................................... 102About Management and Monitoring Utilities........................................................................ 104About Concurrency Control in Greenplum Database...........................................................105About Parallel Data Loading................................................................................................ 113About Redundancy and Failover in Greenplum Database...................................................114About Database Statistics in Greenplum Database............................................................. 116

Managing a Greenplum System..................................................................................................... 124About the Greenplum Database Release Version Number................................................. 124Starting and Stopping Greenplum Database....................................................................... 124Accessing the Database.......................................................................................................127Configuring the Greenplum Database System.....................................................................135Enabling Compression..........................................................................................................137Enabling High Availability and Data Consistency Features................................................. 138Backing Up and Restoring Databases................................................................................. 155Expanding a Greenplum System..........................................................................................193Migrating Data with gpcopy..................................................................................................208Monitoring a Greenplum System..........................................................................................208Routine System Maintenance Tasks....................................................................................224Recommended Monitoring and Maintenance Tasks............................................................ 228

Managing Greenplum Database Access.........................................................................................236Configuring Client Authentication......................................................................................... 236


5

Managing Roles and Privileges............................................................................................264Defining Database Objects..............................................................................................................271

Creating and Managing Databases......................................................................................271Creating and Managing Tablespaces...................................................................................273Creating and Managing Schemas........................................................................................275Creating and Managing Tables............................................................................................ 277Choosing the Table Storage Model..................................................................................... 282Partitioning Large Tables......................................................................................................293Creating and Using Sequences........................................................................................... 306Using Indexes in Greenplum Database............................................................................... 309Creating and Managing Views............................................................................................. 313

Distribution and Skew..................................................................................................................... 314Local (Co-located) Joins.......................................................................................................314Data Skew............................................................................................................................ 314Processing Skew.................................................................................................................. 315

Inserting, Updating, and Deleting Data...........................................................................................318About Concurrency Control in Greenplum Database...........................................................318Inserting Rows...................................................................................................................... 319Updating Existing Rows........................................................................................................320Deleting Rows.......................................................................................................................320Working With Transactions...................................................................................................320Global Deadlock Detector.....................................................................................................322Vacuuming the Database..................................................................................................... 324Running Out of Locks...........................................................................................................324

Querying Data................................................................................................................................. 326About Greenplum Query Processing....................................................................................326About GPORCA....................................................................................................................329Defining Queries................................................................................................................... 342WITH Queries (Common Table Expressions)......................................................................354Using Functions and Operators............................................................................................358Working with JSON Data..................................................................................................... 368Working with XML Data........................................................................................................381Using Full Text Search.........................................................................................................393Using Greenplum MapReduce............................................................................................. 428Query Performance.............................................................................................................. 436Managing Spill Files Generated by Queries........................................................................ 436Query Profiling...................................................................................................................... 436

Working with External Data.............................................................................................................442Accessing External Data with PXF...................................................................................... 442Defining External Tables...................................................................................................... 442Accessing External Data with Foreign Tables..................................................................... 460Using the Greenplum Parallel File Server (gpfdist)..............................................................469

Loading and Unloading Data.......................................................................................................... 473Loading Data Using an External Table................................................................................ 474Loading and Writing Non-HDFS Custom Data.................................................................... 474Handling Load Errors............................................................................................................477Loading Data with gpload.....................................................................................................479Accessing External Data with PXF...................................................................................... 480Transforming External Data with gpfdist and gpload........................................................... 481Loading Data with COPY..................................................................................................... 491Running COPY in Single Row Error Isolation Mode............................................................492Optimizing Data Load and Query Performance................................................................... 492Unloading Data from Greenplum Database......................................................................... 492Formatting Data Files........................................................................................................... 495Example Custom Data Access Protocol.............................................................................. 498

Managing Performance................................................................................................................... 505


6

Defining Database Performance.......................................................................................... 505Common Causes of Performance Issues............................................................................ 506Greenplum Database Memory Overview............................................................................. 509Managing Resources............................................................................................................513Investigating a Performance Problem.................................................................................. 542

Chapter 8: Greenplum Database Security Configuration Guide......... 545Securing the Database....................................................................................................................546Greenplum Database Ports and Protocols..................................................................................... 547Configuring Client Authentication.................................................................................................... 551

Allowing Connections to Greenplum Database....................................................................551Editing the pg_hba.conf File.................................................................................................553Authentication Methods........................................................................................................ 554SSL Client Authentication.....................................................................................................557PAM Based Authentication...................................................................................................559Radius Authentication...........................................................................................................560Limiting Concurrent Connections......................................................................................... 560Encrypting Client/Server Connections..................................................................................561

Configuring Database Authorization................................................................................................562Access Permissions and Roles............................................................................................562Managing Object Privileges..................................................................................................562Using SSH-256 Encryption...................................................................................................563Restricting Access by Time..................................................................................................565Dropping a Time-based Restriction.................................................................................... 567

Greenplum Command Center Security........................................................................................... 568Auditing............................................................................................................................................ 571Encrypting Data and Database Connections.................................................................................. 576

Encrypting gpfdist Connections............................................................................................ 576Encrypting Data at Rest with pgcrypto.................................................................................577

Security Best Practices................................................................................................................... 585

Chapter 10: Greenplum Database Best Practices................................ 589Best Practices Summary.................................................................................................................590System Configuration...................................................................................................................... 596Schema Design............................................................................................................................... 601

Data Types........................................................................................................................... 601Storage Model...................................................................................................................... 601Compression......................................................................................................................... 602Distributions.......................................................................................................................... 603Partitioning............................................................................................................................ 606Indexes..................................................................................................................................608Column Sequence and Byte Alignment............................................................................... 608

Memory and Resource Management with Resource Groups......................................................... 610Memory and Resource Management with Resource Queues........................................................ 613System Monitoring and Maintenance..............................................................................................617

Monitoring............................................................................................................................. 617Updating Statistics with ANALYZE.......................................................................................618Managing Bloat in the Database..........................................................................................619Monitoring Greenplum Database Log Files..........................................................................623

Loading Data................................................................................................................................... 625INSERT Statement with Column Values..............................................................................625COPY Statement.................................................................................................................. 625External Tables.....................................................................................................................625External Tables with Gpfdist................................................................................................ 625


7

Gpload...................................................................................................................................626Best Practices.......................................................................................................................627

Security............................................................................................................................................ 628Encrypting Data and Database Connections.................................................................................. 631Tuning SQL Queries....................................................................................................................... 640

How to Generate Explain Plans........................................................................................... 640How to Read Explain Plans................................................................................................. 640Optimizing Greenplum Queries............................................................................................ 642

High Availability............................................................................................................................... 644Disk Storage......................................................................................................................... 644Master Mirroring....................................................................................................................644Segment Mirroring................................................................................................................ 645Dual Clusters........................................................................................................................ 646Backup and Restore.............................................................................................................646Detecting Failed Master and Segment Instances................................................................ 647Segment Mirroring Configuration..........................................................................................648

Chapter 12: Greenplum Database Utility Guide................................... 653About the Greenplum Database Utilities.........................................................................................654

Referencing IP Addresses....................................................................................................654Running Backend Server Programs.....................................................................................654

Utility Reference.............................................................................................................................. 656analyzedb..............................................................................................................................656clusterdb................................................................................................................................660createdb................................................................................................................................ 662createlang............................................................................................................................. 664createuser............................................................................................................................. 666dropdb................................................................................................................................... 668droplang................................................................................................................................ 670dropuser................................................................................................................................ 671gpactivatestandby................................................................................................................. 672gpaddmirrors......................................................................................................................... 674gpbackup...............................................................................................................................677gpcheckcat............................................................................................................................ 683gpcheckperf...........................................................................................................................686gpconfig.................................................................................................................................689gpcopy...................................................................................................................................692gpdeletesystem..................................................................................................................... 692gpexpand.............................................................................................................................. 693gpfdist....................................................................................................................................696gpinitstandby......................................................................................................................... 700gpinitsystem.......................................................................................................................... 702gpload................................................................................................................................... 709gplogfilter...............................................................................................................................719gpmapreduce........................................................................................................................ 722gpmapreduce.yaml................................................................................................................724gpmovemirrors...................................................................................................................... 731gpperfmon_install..................................................................................................................732gppkg.................................................................................................................................... 736gprecoverseg........................................................................................................................ 738gpreload................................................................................................................................ 742gprestore............................................................................................................................... 744gpscp.....................................................................................................................................749gpssh.....................................................................................................................................751gpssh-exkeys........................................................................................................................ 754


8

gpstart................................................................................................................................... 756gpstate.................................................................................................................................. 758gpstop................................................................................................................................... 762gpsys1...................................................................................................................................765pg_config...............................................................................................................................765pg_dump............................................................................................................................... 767pg_dumpall............................................................................................................................776pg_restore............................................................................................................................. 780pgbouncer............................................................................................................................. 785pgbouncer.ini.........................................................................................................................787pgbouncer-admin.................................................................................................................. 799psql........................................................................................................................................807reindexdb.............................................................................................................................. 832vacuumdb..............................................................................................................................834

Additional Supplied Programs......................................................................................................... 837

Chapter 14: Greenplum Database Reference Guide............................ 838SQL Commands.............................................................................................................................. 839

SQL Syntax Summary..........................................................................................................842ABORT..................................................................................................................................879ALTER AGGREGATE...........................................................................................................879ALTER COLLATION.............................................................................................................881ALTER CONVERSION......................................................................................................... 882ALTER DATABASE.............................................................................................................. 883ALTER DEFAULT PRIVILEGES.......................................................................................... 884ALTER DOMAIN...................................................................................................................887ALTER EXTENSION.............................................................................................................889ALTER EXTERNAL TABLE..................................................................................................891ALTER FOREIGN DATA WRAPPER...................................................................................893ALTER FOREIGN TABLE.................................................................................................... 894ALTER FUNCTION...............................................................................................................898ALTER GROUP.................................................................................................................... 900ALTER INDEX...................................................................................................................... 901ALTER LANGUAGE............................................................................................................. 903ALTER OPERATOR............................................................................................................. 903ALTER OPERATOR CLASS................................................................................................ 904ALTER OPERATOR FAMILY...............................................................................................905ALTER PROTOCOL............................................................................................................. 908ALTER RESOURCE GROUP.............................................................................................. 909ALTER RESOURCE QUEUE...............................................................................................911ALTER ROLE....................................................................................................................... 913ALTER SCHEMA..................................................................................................................918ALTER SEQUENCE............................................................................................................. 919ALTER SERVER.................................................................................................................. 921ALTER TABLE......................................................................................................................922ALTER TABLESPACE..........................................................................................................935ALTER TEXT SEARCH CONFIGURATION.........................................................................936ALTER TEXT SEARCH DICTIONARY................................................................................ 937ALTER TEXT SEARCH PARSER........................................................................................939ALTER TEXT SEARCH TEMPLATE....................................................................................939ALTER TYPE........................................................................................................................940ALTER USER....................................................................................................................... 943ALTER USER MAPPING..................................................................................................... 944ALTER VIEW........................................................................................................................ 944ANALYZE..............................................................................................................................946


9

BEGIN................................................................................................................................... 950CHECKPOINT.......................................................................................................................952CLOSE.................................................................................................................................. 952CLUSTER............................................................................................................................. 953COMMENT............................................................................................................................955COMMIT................................................................................................................................958COPY.................................................................................................................................... 958CREATE AGGREGATE........................................................................................................970CREATE CAST.....................................................................................................................977CREATE COLLATION..........................................................................................................980CREATE CONVERSION...................................................................................................... 981CREATE DATABASE........................................................................................................... 982CREATE DOMAIN................................................................................................................984CREATE EXTENSION..........................................................................................................986CREATE EXTERNAL TABLE...............................................................................................988CREATE FOREIGN DATA WRAPPER................................................................................996CREATE FOREIGN TABLE................................................................................................. 998CREATE FUNCTION..........................................................................................................1000CREATE GROUP............................................................................................................... 1008CREATE INDEX................................................................................................................. 1009CREATE LANGUAGE........................................................................................................ 1013CREATE OPERATOR........................................................................................................ 1016CREATE OPERATOR CLASS........................................................................................... 1020CREATE OPERATOR FAMILY..........................................................................................1024CREATE PROTOCOL........................................................................................................ 1025CREATE RESOURCE GROUP......................................................................................... 1026CREATE RESOURCE QUEUE..........................................................................................1029CREATE ROLE.................................................................................................................. 1032CREATE RULE...................................................................................................................1037CREATE SCHEMA.............................................................................................................1039CREATE SEQUENCE........................................................................................................ 1040CREATE SERVER............................................................................................................. 1043CREATE TABLE.................................................................................................................1044CREATE TABLE AS...........................................................................................................1058CREATE TABLESPACE.....................................................................................................1061CREATE TEXT SEARCH CONFIGURATION....................................................................1063CREATE TEXT SEARCH DICTIONARY............................................................................1064CREATE TEXT SEARCH PARSER...................................................................................1065CREATE TEXT SEARCH TEMPLATE...............................................................................1066CREATE TYPE...................................................................................................................1067CREATE USER.................................................................................................................. 1075CREATE USER MAPPING................................................................................................ 1076CREATE VIEW................................................................................................................... 1076DEALLOCATE.................................................................................................................... 1079DECLARE........................................................................................................................... 1080DELETE.............................................................................................................................. 1082DISCARD............................................................................................................................ 1085DO.......................................................................................................................................1086DROP AGGREGATE..........................................................................................................1087DROP CAST.......................................................................................................................1089DROP COLLATION............................................................................................................ 1089DROP CONVERSION........................................................................................................ 1090DROP DATABASE............................................................................................................. 1091DROP DOMAIN.................................................................................................................. 1091DROP EXTENSION............................................................................................................1092DROP EXTERNAL TABLE.................................................................................................1093


10

DROP FOREIGN DATA WRAPPER..................................................................................1094DROP FOREIGN TABLE................................................................................................... 1095DROP FUNCTION..............................................................................................................1095DROP GROUP................................................................................................................... 1096DROP INDEX..................................................................................................................... 1097DROP LANGUAGE.............................................................................................................1098DROP OPERATOR............................................................................................................ 1098DROP OPERATOR CLASS............................................................................................... 1099DROP OPERATOR FAMILY.............................................................................................. 1100DROP OWNED...................................................................................................................1101DROP PROTOCOL............................................................................................................ 1102DROP RESOURCE GROUP..............................................................................................1103DROP RESOURCE QUEUE.............................................................................................. 1104DROP ROLE.......................................................................................................................1105DROP RULE.......................................................................................................................1106DROP SCHEMA................................................................................................................. 1106DROP SEQUENCE............................................................................................................ 1107DROP SERVER..................................................................................................................1108DROP TABLE..................................................................................................................... 1109DROP TABLESPACE.........................................................................................................1109DROP TEXT SEARCH CONFIGURATION........................................................................1110DROP TEXT SEARCH DICTIONARY................................................................................1111DROP TEXT SEARCH PARSER....................................................................................... 1112DROP TEXT SEARCH TEMPLATE...................................................................................1112DROP TYPE....................................................................................................................... 1113DROP USER...................................................................................................................... 1114DROP USER MAPPING.....................................................................................................1114DROP VIEW....................................................................................................................... 1115END.....................................................................................................................................1116EXECUTE........................................................................................................................... 1116EXPLAIN............................................................................................................................. 1117FETCH................................................................................................................................ 1122GRANT................................................................................................................................1125INSERT............................................................................................................................... 1130LOAD.................................................................................................................................. 1132LOCK.................................................................................................................................. 1133MOVE..................................................................................................................................1136PREPARE........................................................................................................................... 1138REASSIGN OWNED...........................................................................................................1139REINDEX............................................................................................................................ 1140RELEASE SAVEPOINT......................................................................................................1142RESET................................................................................................................................ 1142REVOKE............................................................................................................................. 1143ROLLBACK......................................................................................................................... 1146ROLLBACK TO SAVEPOINT.............................................................................................1147SAVEPOINT........................................................................................................................1148SELECT.............................................................................................................................. 1149SELECT INTO.................................................................................................................... 1167SET..................................................................................................................................... 1169SET CONSTRAINTS.......................................................................................................... 1170SET ROLE.......................................................................................................................... 1171SET SESSION AUTHORIZATION..................................................................................... 1173SET TRANSACTION.......................................................................................................... 1174SHOW................................................................................................................................. 1177START TRANSACTION..................................................................................................... 1178TRUNCATE.........................................................................................................................1179


11

UPDATE..............................................................................................................................1181VACUUM.............................................................................................................................1184VALUES.............................................................................................................................. 1187

Data Types.................................................................................................................................... 1190Date/Time Types................................................................................................................ 1192Pseudo-Types..................................................................................................................... 1202Text Search Data Types.................................................................................................... 1204Range Types...................................................................................................................... 1206

Summary of Built-in Functions...................................................................................................... 1211Greenplum Database Function Types................................................................................1211Built-in Functions and Operators........................................................................................1212JSON Functions and Operators......................................................................................... 1215Window Functions.............................................................................................................. 1222Advanced Aggregate Functions......................................................................................... 1224Text Search Functions and Operators............................................................................... 1226Range Functions and Operators........................................................................................ 1230

Procedural Languages and Analytics Extensions.........................................................................1233PL/Container Language......................................................................................................1233PL/Java Language..............................................................................................................1257PL/Perl Language............................................................................................................... 1272PL/pgSQL Language.......................................................................................................... 1277PL/Python Language.......................................................................................................... 1284PL/R Language...................................................................................................................1293MADlib Extension for Analytics.......................................................................................... 1299PostGIS Geospatial Extension........................................................................................... 1309

Additional Supplied Modules......................................................................................................... 1316auto_explain........................................................................................................................1316citext....................................................................................................................................1316dblink...................................................................................................................................1317diskquota.............................................................................................................................1320fuzzystrmatch...................................................................................................................... 1324gp_sparse_vector................................................................................................................1324hstore.................................................................................................................................. 1328orafce.................................................................................................................................. 1329pageinspect.........................................................................................................................1331pgcrypto.............................................................................................................................. 1331sslinfo.................................................................................................................................. 1332

Character Set Support...................................................................................................................1333Setting the Character Set...................................................................................................1335Character Set Conversion Between Server and Client...................................................... 1335

Server Configuration Parameters..................................................................................................1338Parameter Types and Values.............................................................................................1338Setting Parameters............................................................................................................. 1338Parameter Categories.........................................................................................................1339Configuration Parameters...................................................................................................1349

System Catalogs........................................................................................................................... 1429System Tables.................................................................................................................... 1429System Views..................................................................................................................... 1430System Catalogs Definitions...............................................................................................1431

The gp_toolkit Administrative Schema..........................................................................................1530Checking for Tables that Need Routine Maintenance........................................................1530Checking for Locks.............................................................................................................1531Checking Append-Optimized Tables.................................................................................. 1533Viewing Greenplum Database Server Log Files................................................................ 1537Checking Server Configuration Files..................................................................................1540Checking for Failed Segments........................................................................................... 1541


12

Checking Resource Group Activity and Status.................................................................. 1542Checking Resource Queue Activity and Status................................................................. 1546Checking Query Disk Spill Space Usage...........................................................................1548Viewing Users and Groups (Roles)....................................................................................1550Checking Database Object Sizes and Disk Space............................................................ 1550Checking for Uneven Data Distribution.............................................................................. 1554

The gpperfmon Database..............................................................................................................1556database_*.........................................................................................................................1558diskspace_*....................................................................................................................... 1559interface_stats_*................................................................................................................ 1559log_alert_*..........................................................................................................................1561queries_*............................................................................................................................. 1562segment_*..........................................................................................................................1564socket_stats_*.....................................................................................................................1565system_*............................................................................................................................. 1566dynamic_memory_info........................................................................................................ 1568memory_info...................................................................................................................... 1568

Server Programmatic Interfaces....................................................................................................1570Greenplum Partner Connector API.................................................................................... 1570Developing a Background Worker Process....................................................................... 1589

SQL Features, Reserved and Key Words, and Compliance........................................................ 1592Summary of Greenplum Features...................................................................................... 1592Reserved Identifiers and SQL Key Words......................................................................... 1603SQL 2008 Optional Feature Compliance........................................................................... 1619

Chapter 16: Greenplum Client and Loader Tools Package............... 1648

Chapter 17: About the Tools Package................................................ 1649

Chapter 18: Installing the Client and Loader Tools Package............ 1650Installation Procedure....................................................................................................................1651About Your Installation.................................................................................................................. 1652Running the UNIX Tools Installer................................................................................................. 1653

Prerequisites....................................................................................................................... 1653Procedure............................................................................................................................1653

Running the Windows Tools Installer........................................................................................... 1654Prerequisites....................................................................................................................... 1654Procedure............................................................................................................................1654

Chapter 19: Configuring Greenplum Database for Remote ClientAccess................................................................................................. 1655

Chapter 20: Configuring a Client System for KerberosAuthentication.................................................................................... 1656

Chapter 21: Using the Client and Loader Tools................................. 1657Prerequisites.................................................................................................................................. 1658Setting Up Your Greenplum Database Clients Runtime Environment.......................................... 1659Running the Client and Loader Programs.................................................................................... 1660


13

Greenplum Database Documentation References........................................................................1661Windows Considerations............................................................................................................... 1662

Chapter 22: Client and Loader Utility Reference................................1663

Chapter 23: DataDirect ODBC Drivers for Pivotal Greenplum...........1664Prerequisites.................................................................................................................................. 1665Supported Client Platforms........................................................................................................... 1666Installing on Linux Systems.......................................................................................................... 1667

Configuring the Driver on Linux......................................................................................... 1668Testing the Driver Connection on Linux.............................................................................1669

Installing on Windows Systems.................................................................................................... 1671Verifying the Version on Windows..................................................................................... 1671Configuring and Testing the Driver on Windows................................................................1671

DataDirect Driver Documentation..................................................................................................1673

Chapter 24: DataDirect JDBC Driver for Pivotal Greenplum............. 1674Prerequisites.................................................................................................................................. 1675Downloading the DataDirect JDBC Driver.................................................................................... 1676Obtaining Version Details for the Driver....................................................................................... 1677Usage Information......................................................................................................................... 1678DataDirect Driver Documentation..................................................................................................1679

Pivotal Greenplum 6.1 Release Notes Release Notes

14

Chapter 3

Pivotal Greenplum 6.1 Release Notes

This document contains pertinent release information about Pivotal Greenplum Database 6.1 releases.For previous versions of the release notes for Greenplum Database, go to Pivotal Greenplum DatabaseDocumentation. For information about Greenplum Database end of life, see Pivotal Greenplum Databaseend of life policy.

Pivotal Greenplum 6 software is available for download from the Pivotal Greenplum page on PivotalNetwork.

Pivotal Greenplum 6 is based on the open source Greenplum Database project code.

Important: Pivotal Support does not provide support for open source versions of GreenplumDatabase. Only Pivotal Greenplum Database is supported by Pivotal Support.

https://gpdb.docs.pivotal.io/https://gpdb.docs.pivotal.io/https://support.pivotal.io/hc/en-us/articles/201143553-GPDB-software-EOL-policyhttps://support.pivotal.io/hc/en-us/articles/201143553-GPDB-software-EOL-policyhttps://network.pivotal.io/products/pivotal-gpdbhttps://network.pivotal.io/products/pivotal-gpdbhttp://greenplum.org/


15

Release 6.1.1Release Date: 2019-11-15

Pivotal Greenplum 6.1.1 is a maintenance release that resolves several issues.

Resolved IssuesPivotal Greenplum 6.1.1 resolves these issues:

8590 - Postgres Planner

A query that used the Postgres planner could return incorrect results if it specified a volatilefunction in a LIMIT clause (for example, LIMIT (random() * 10)). This occurredbecause Greenplum evaluated the LIMIT clause separately on each segment instance toobtain a preliminary limit, before evaluating it once again as the query was dispatched. Theproblem was fixed by ensuring that a volatile functions in a LIMIT clause functions are notpushed to segment instances for evaluation.

30083 - Postgres Planner

Fixed a problem in the Postgres planner that could result in the error variable notfound in subplan target list for certain queries.

30227 - Server

Greenplum Database with resource groups enabled might generate a PANIC when usingan extension with improper debug_query_string settings. The cause was a messagecontext issue and it has been resolved.


16

Release 6.1.0Release Date: 2019-11-1

Pivotal Greenplum 6.1.0 is a minor release that includes new features and resolves several issues.

New FeaturesGreenplum Database 6.1.0 includes these new features:

• Greenplum Stream Server 1.3 is included, which introduces new features and bug fixes. New GPSSfeatures include:

• GPSS now supports log rotation, utilizing a mechanism that you can easily integrate with the Linuxlogrotate system. See Managing GPSS Log Files for more information.

• GPSS has added the new INPUT:FILTER load configuration property. This property enables you tospecify a filter that GPSS applies to Kafka input data before loading it into Greenplum Database.

• GPSS displays job progress by partition when you provide the --partition flag to the gpsscliprogress command.

• GPSS enables you to load Kafka data that was emitted since a specific timestamp into GreenplumDatabase. To use this feature, you provide the --force-reset-timestamp flag when you rungpsscli load, gpsscli start, or gpkafka load.

• GPSS now supports update and merge operations on data stored in a Greenplum Databasetable. The load configuration file accepts MODE, MATCH_COLUMNS, UPDATE_COLUMNS, andUPDATE_CONDITION property values to direct these operations. Example: Merging Data from Kafkainto Greenplum Using the Greenplum Stream Server provides an example merge scenario.

• GPSS supports Kerberos authentication to both Kafka and Greenplum Database.• GPSS supports SSL encryption between GPSS and Kafka.• GPSS supports SSL encryption on the data channel between GPSS and Greenplum Database.

• The DataDirect JDBC and ODBC drivers were updated to versions 5.1.4.000270 (F000450.U000214)and 07.16.0334 (B0510, U0363), respectively.

The DataDirect JDBC driver introduces support for the prepareThreshold connection parameter,which specifies the number of prepared statement executions that can be performed before the driverswitches to using server-side prepared statements. This parameter defaults to 0, which preserves theearlier driver behavior of always using server-side prepare for prepared statements. Set a numbergreater than 1 to set a threshold after which server-side prepare is used.

Note: ExecuteBatch() always uses server-side prepare for prepared statements. Thismatches the behavior of the Postgres open source driver.

When the prepareThreshold value is greater than 1, parameterized operations do not send any SQLprepare calls with connection.prepareStatement(). The driver instead sends the query all atonce, at execution time. Because of this limitation, the driver must determine the type of every columnusing the JDBC API before sending the query to the server. This determination works for many datatypes, but does not work for the following types that could be mapped to multiple Greenplum data types:

• BIT VARYING• BOOLEAN• JSON• TIME WITH TIME ZONE• UUIDCOL

You must set prepareThreshold to 0 before using parameterized operations with any of the abovetypes. Examine the ResultSetMetaData object in advance to determine if any of the above types

../greenplum-stream/instcfgmgt.html#logfiles


17

are used in a query. Also keep in mind that GPORCA does not support prepared statements that haveparameterized values, and will fall back to using the Postgres Planner.

Resolved IssuesPivotal Greenplum 6.1.0 is a minor release that resolves these issues:

8804 - Server

In some cases, running the EXPLAIN ANALYZE command on a sorted query in utilitymode would cause the segment to crash. This issue is fixed. Greenplum Database nolonger crashes in this situation.

8636 - Server

Some users encountered Error: unrecognized parameter "appendoptimized"while creating a partitioned table that specified the appendoptimized=true storageparameter. This issue is fixed; the Greenplum Database server now properly recognizesthe appendoptimized parameter when it is specified on partition table creation.

26225 - gpcheckcat

The gpcheckcat utility failed to generate a summary report if there was an orphanTOAST table entry in one of the segments. This is fixed. The string "N/A" is reported whenthere is no relation OID to report.

29580 - Management and Monitoring

During Greenplum Database startup, an extra empty log file was produced aheadof the current date while performing time-based rotation of log files. For example,if Greenplum started at midnight September 2nd, two log files were generated,gpdb-2019-09-02_000000.csv and gpdb-2019-09-03_000000.csv. This issuehas now been fixed.

29984 - Server

During startup, idle query executor (QE) processes can commit up to 16MB of memoryeach, but they are not tracked by the Linux virtual memory tracker. In a worst-casescenario, these idle processes could trigger OOM errors that were difficult to diagnose. Toprevent these situations, Greenplum now hard-codes a startup memory cost to account foruntracked QE processes.

30112 - Query Optimizer

For some queries against partitioned tables that contain a large amount of data, GPORCAgenerated a sub-optimal query plan because of inaccurate cardinality estimation. Thisissue has been resolved. GPORCA cardinality estimation has been improved.

30183, 30184 - analyzedb

When running the analyzedb command with the --skip_root_stats option, thecommand could take a long time to finish when analyzing a partitioned table with manypartitions due to how the root partition statistics were handled when the partitions wereanalyzed. This issue has been resolved. Now, only partition statistics are updated.

Note: GPORCA uses root partition statistics. If you use --skip_root_stats option, you should ensure that root partition statistics areup to date so that GPORCA does not produce inferior query plans due to staleroot partition statistics.

30149 - Query Execution

A query might fail and return an error with the message invalid seek insequential BufFile when the server configuration file gp_workfile_compressionis on and the query spills to temporary workfiles. The error was caused due to an issueworking with workfiles that contain compressed data. The issue has been resolved bycorrectly handling the compressed workfile data.

https://github.com/greenplum-db/gpdb/issues/8804https://github.com/greenplum-db/gpdb/issues/8636


18

30150 - Query Execution

A query might fail and return with the message AssignTransactionId()called by Segment Reader process when the server configuration parametertemp_tablespaces is set. The error was cause by an internal locking and transaction IDissue. This issue has been resolved by removing the requirement to acquire the lock.


GPORCA might return incorrect results when a the query contains a join predicate whereone side is distributed on a citext column, and the other is not. GPORCA did not usethe correct hash when generating a plan that redistributes the citext column. NowGreenplum Database falls back to the Postgres Planner for the specified type of query.

30183 - analyzedb

The analyzedb command could take a long time to finish when analyzing a table withmany partitions. The command's performance has been greatly improved by waiting toupdate the root partition statistics until all leaf partitions of a table have been analyzed.

164823612 - gpss

GPSS incorrectly treated Kafka jobs that specified the same Kafka topic and Greenplumoutput schema name and output table name, but different database names, as the samejob. This issue has been resolved. GPSS now includes the Greenplum database namewhen constructing a job definition.

167997441 - gpss

GPSS did not save error data to the external table error log when it encountered anincorrectly-formatted JSON or Avro message. This issue has been fixed; invokinggp_read_error_log() on the external table now displays the offending data.

168130147 - gpss

In some situations, specifying the --force-reset-earliest flag when loading datafailed to read from the correct offset. This problem has been fixed. (Using the --force-reset-xxx flags outside of an offset mismatch scenario is discouraged.)


Certain queries with btree indexes on Append Optimized (AO) tables were unnecessarilyslow due to GPORCA selecting a scan with high transformation and cost impact. Thisissue has been fixed by improving GPORCA handling of btree type indexes.


In some situations, a query ran slow because GPORCA did not produce an optimal planwhen it encountered a null-rejecting predicate where an operand could be false or null, butnot true. This issue is fixed; GPORCA now produces a more optimal plan when evaluatingnull-rejecting predicates for AND and OR operands.


For certain queries with a UNION operator over a large number of children, GPORCAquery optimization required a long time. This issue has been addressed by adding theability to derive scalar properties on demand.


Some queries in GPORCA were consuming more memory than necessary due tosuboptimal memory tracking. This has been fixed by optimizing memory accounting insideGPORCA.

169081574 - Interconnect

Greenplum Database might generate a PANIC when the server configuration parametergp_interconnect_type is TCP due to an issue with memory management duringinterconnect setup. The issue has been resolved by properly managing the internalinterconnect object memory.


19

169117536 - Execution

Greenplum Database might generate a PANIC when the server configuration parameterlog_min_messages is set to debug5. Greenplum Database did not properly handle adebug5 message correctly. The issue is resolved.

169198230 - Plan Cache

A prepared statement might run slow because a cost model issue prevented GreenplumDatabase from generating a direct dispatch plan for the statement. This issue is fixed.Greenplum Database now introduces non-direct dispatch cost into the cost model only forcached plans, and tries to use direct dispatch for prepared statements when possible.


20

Upgrading to Greenplum 6.1Note: Greenplum 6 does not support direct upgrades from Greenplum 4 or Greenplum 5 releases,or from earlier Greenplum 6 Beta releases.

See Upgrading from an Earlier Greenplum 6 Release to upgrade your existing Greenplum 6.x software toGreenplum 6.1.


21

Deprecated FeaturesDeprecated features will be removed in a future major release of Greenplum Database. Pivotal Greenplum6.x deprecates:

• The server configuration parameter gp_ignore_error_table (deprecated since 6.0).

To avoid a Greenplum Database syntax error, set the value of this parameter to true when yourun applications that execute CREATE EXTERNAL TABLE or COPY commands that include the nowremoved Greenplum Database 4.3.x INTO ERROR TABLE clause.

• Specifying => as an operator name in the CREATE OPERATOR command (deprecated since 6.0).• The Greenplum external table C API (deprecated since 6.0).

Any developers using this API are encouraged to use the new Foreign Data Wrapper API in its place.• Commas placed between a SUBPARTITION TEMPLATE clause and its corresponding SUBPARTITION

BY clause, and between consecutive SUBPARTITION BY clauses in a CREATE TABLE command(deprecated since 6.0).

Using this undocumented syntax will generate a deprecation warning message.• The timestamp format YYYYMMDDHH24MISS (deprecated since 6.0).

This format could not be parsed unambiguously in previous Greenplum Database releases, and is notsupported in PostgreSQL 9.4.

• The createlang and droplang utilities (deprecated since 6.0).• The pg_resqueue_status system view (deprecated since 6.0).

Use the gp_toolkit.gp_resqueue_status view instead.• The GLOBAL and LOCAL modifiers when creating a temporary table with the CREATE TABLE and

CREATE TABLE AS commands (deprecated since 6.0).

These keywords are present for SQL standard compatibility, but have no effect in Greenplum Database.• The Greenplum Platform Extension Framework (PXF) HDFS profile names for the Text, Avro, JSON,

Parquet, and SequenceFile data formats (deprecated since 5.16).

Refer to Connectors, Data Formats, and Profiles in the PXF Hadoop documentation for moreinformation.

• Using WITH OIDS or oids=TRUE to assign an OID system column when creating or altering a table(deprecated since 6.0).

• Allowing superusers to specify the SQL_ASCII encoding regardless of the locale settings (deprecatedsince 6.0).

This choice may result in misbehavior of character-string functions when data that is not encoding-compatible with the locale is stored in the database.

• The @@@ text search operator (deprecated since 6.0).

This operator is currently a synonym for the @@ operator.• The unparenthesized syntax for option lists in the VACUUM command (deprecated since 6.0).

This syntax requires that the options to the command be specified in a specific order.• The plain pgbouncer authentication type (auth_type = plain) (deprecated since 4.x).

../pxf/access_hdfs.html#hadoop_connectors


22

Migrating Data to Greenplum 6Note: Greenplum 6 does not support direct upgrades from Greenplum 4 or Greenplum 5 releases,or from earlier Greenplum 6 Beta releases.

See Migrating Data from Greenplum 4.3 or 5 for guidelines and considerations for migrating existingGreenplum data to Greenplum 6, using standard backup and restore procedures.


23

Known Issues and LimitationsPivotal Greenplum 6 has these limitations:

• Upgrading a Greenplum Database 4 or 5 release, or Greenplum 6 Beta release, to Pivotal Greenplum 6is not supported.

• MADlib, GPText, and PostGIS are not yet provided for installation on Ubuntu systems.• gpcopy cannot yet copy data from Greenplum 4 or 5 to Greenplum 6.• Greenplum 6 is not supported for installation on DCA systems.• Greenplum for Kubernetes is not yet provided with this release.

The following table lists key known issues in Pivotal Greenplum 6.x.

Table 1: Key Known Issues in Pivotal Greenplum 6.x

Issue Category Description

169200795 GreenplumStream Server

When loading Kafka data into Greenplum Database in UPDATE andMERGE modes, GPSS requires that a MAPPING exist for each columnname identified in the MATCH_COLUMNS and UPDATE_COLUMNS lists.

168548176 gpbackup When using gpbackup to back up a Greenplum Database 5.7.1 orearlier 5.x release with resource groups enabled, gpbackup returnsa column not found error for t6.value AS memoryauditor.

164791118 PL/R PL/R cannot be installed using the deprecated createlang utility,and displays the error:

createlang: language installation failed: ERROR: no schema has been selected to create in

Worka

Documents

Pivotal Greenplum DatabaseContents Release Notes 3 Contents Chapter 3: Pivotal Greenplum 6.1 Release Notes.....14 Release 6.1.1..... 15 Resolved