Pivotal Greenplum Database · Contents Release Notes 6 Managing Performance..... 507

PRODUCT DOCUMENTATION

Pivotal™ GreenplumDatabase®Version 6.2

Pivotal Greenplum DatabaseDocumentationRev: A02

© 2019 Pivotal Software, Inc.

Copyright Release Notes

2

Notice

Copyright

Privacy Policy | Terms of Use

Copyright © 2019 Pivotal Software, Inc. All rights reserved.

Pivotal Software, Inc. believes the information in this publication is accurate as of its publication date. Theinformation is subject to change without notice. THE INFORMATION IN THIS PUBLICATION IS PROVIDED"AS IS." PIVOTAL SOFTWARE, INC. ("Pivotal") MAKES NO REPRESENTATIONS OR WARRANTIES OF ANYKIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMSIMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Use, copying, and distribution of any Pivotal software described in this publication requires an applicablesoftware license.

All trademarks used herein are the property of Pivotal or their respective owners.

Revised December 2019 (6.2.1)

http://pivotal.io/privacy-policyhttp://pivotal.io/terms-of-use

Contents Release Notes

3

Contents

Chapter 2: Pivotal Greenplum 6.2 Release Notes.................................. 14Release 6.2.1.................................................................................................................................... 15

New Features......................................................................................................................... 15Changed Features.................................................................................................................. 16Resolved Issues..................................................................................................................... 16Upgrading to Greenplum 6.2.1...............................................................................................18

Deprecated Features.........................................................................................................................19Migrating Data to Greenplum 6........................................................................................................ 21Known Issues and Limitations.......................................................................................................... 22Differences Compared to Open Source Greenplum Database.........................................................23

Chapter 3: Installing and Upgrading Greenplum................................... 24Platform Requirements......................................................................................................................25

Operating Systems................................................................................................................. 25Hardware and Network...........................................................................................................27Storage....................................................................................................................................27Tools and Extensions Compatibility....................................................................................... 28Hadoop Distributions.............................................................................................................. 30

Introduction to Greenplum.................................................................................................................31The Greenplum Master...........................................................................................................32The Segments........................................................................................................................ 32The Interconnect.....................................................................................................................36ETL Hosts for Data Loading.................................................................................................. 38Greenplum Performance Monitoring...................................................................................... 39

Estimating Storage Capacity............................................................................................................. 41Calculating Usable Disk Capacity.......................................................................................... 41Calculating User Data Size.................................................................................................... 42Calculating Space Requirements for Metadata and Logs......................................................42

Configuring Your Systems.................................................................................................................43Disabling SELinux and Firewall Software.............................................................................. 43Recommended OS Parameters Settings............................................................................... 44Synchronizing System Clocks................................................................................................ 51Creating the Greenplum Administrative User.........................................................................52Next Steps.............................................................................................................................. 53

Installing the Greenplum Database Software................................................................................... 54Installing Greenplum Database.............................................................................................. 54Enabling Passwordless SSH.................................................................................................. 55Confirming Your Installation................................................................................................... 56About Your Greenplum Database Installation........................................................................56Next Steps.............................................................................................................................. 56

Creating the Data Storage Areas......................................................................................................57Creating Data Storage Areas on the Master and Standby Master Hosts...............................57Creating Data Storage Areas on Segment Hosts.................................................................. 57Next Steps.............................................................................................................................. 58

Validating Your Systems................................................................................................................... 59Validating Network Performance............................................................................................ 59Validating Disk I/O and Memory Bandwidth...........................................................................60

Initializing a Greenplum Database System.......................................................................................61


4

Overview................................................................................................................................. 61Initializing Greenplum Database.............................................................................................61Setting Greenplum Environment Variables............................................................................ 64Next Steps.............................................................................................................................. 65

Installing Optional Extensions........................................................................................................... 67Procedural Language, Machine Learning, and Geospatial Extensions..................................67Python Data Science Module Package..................................................................................67R Data Science Library Package........................................................................................... 71Greenplum Platform Extension Framework (PXF)................................................................. 74

Installing Additional Supplied Modules..............................................................................................75Configuring Timezone and Localization Settings..............................................................................76

Configuring the Timezone...................................................................................................... 76About Locale Support in Greenplum Database..................................................................... 76Character Set Support............................................................................................................78Setting the Character Set.......................................................................................................80Character Set Conversion Between Server and Client..........................................................81

Upgrading from an Earlier Greenplum 6 Release............................................................................ 84Upgrading from 6.x to a Newer 6.x Release......................................................................... 84Troubleshooting a Failed Upgrade.........................................................................................86

Migrating Data from Greenplum 4.3 or 5..........................................................................................87Preparing the Greenplum 6 Cluster....................................................................................... 87Preparing Greenplum 4.3 and 5 Databases for Backup........................................................ 88Backing Up and Restoring a Database..................................................................................90Completing the Migration........................................................................................................91

Enabling iptables (Optional).............................................................................................................. 93Example iptables Rules..........................................................................................................93

Installation Management Utilities.......................................................................................................96Greenplum Environment Variables................................................................................................... 97

Required Environment Variables............................................................................................97Optional Environment Variables............................................................................................. 97

Example Ansible Playbook................................................................................................................99

Chapter 4: Greenplum Database Administrator Guide........................ 101Greenplum Database Concepts...................................................................................................... 102

About the Greenplum Architecture....................................................................................... 102About Management and Monitoring Utilities........................................................................ 104About Concurrency Control in Greenplum Database...........................................................105About Parallel Data Loading................................................................................................ 113About Redundancy and Failover in Greenplum Database...................................................114About Database Statistics in Greenplum Database............................................................. 116

Managing a Greenplum System..................................................................................................... 124About the Greenplum Database Release Version Number................................................. 124Starting and Stopping Greenplum Database....................................................................... 124Accessing the Database.......................................................................................................127Configuring the Greenplum Database System.....................................................................135Enabling Compression..........................................................................................................137Enabling High Availability and Data Consistency Features................................................. 138Backing Up and Restoring Databases................................................................................. 155Expanding a Greenplum System..........................................................................................194Migrating Data with gpcopy..................................................................................................209Monitoring a Greenplum System..........................................................................................209Routine System Maintenance Tasks....................................................................................225Recommended Monitoring and Maintenance Tasks............................................................ 229

Managing Greenplum Database Access.........................................................................................237Configuring Client Authentication......................................................................................... 237


5

Managing Roles and Privileges............................................................................................265Defining Database Objects..............................................................................................................272

Creating and Managing Databases......................................................................................272Creating and Managing Tablespaces...................................................................................274Creating and Managing Schemas........................................................................................276Creating and Managing Tables............................................................................................ 278Choosing the Table Storage Model..................................................................................... 283Partitioning Large Tables......................................................................................................294Creating and Using Sequences........................................................................................... 307Using Indexes in Greenplum Database............................................................................... 310Creating and Managing Views............................................................................................. 314Creating and Managing Materialized Views.........................................................................314

Distribution and Skew..................................................................................................................... 316Local (Co-located) Joins.......................................................................................................316Data Skew............................................................................................................................ 316Processing Skew.................................................................................................................. 317

Inserting, Updating, and Deleting Data...........................................................................................320About Concurrency Control in Greenplum Database...........................................................320Inserting Rows...................................................................................................................... 321Updating Existing Rows........................................................................................................322Deleting Rows.......................................................................................................................322Working With Transactions...................................................................................................323Global Deadlock Detector.....................................................................................................324Vacuuming the Database..................................................................................................... 326Running Out of Locks...........................................................................................................326

Querying Data................................................................................................................................. 328About Greenplum Query Processing....................................................................................328About GPORCA....................................................................................................................331Defining Queries................................................................................................................... 344WITH Queries (Common Table Expressions)......................................................................356Using Functions and Operators............................................................................................360Working with JSON Data..................................................................................................... 370Working with XML Data........................................................................................................383Using Full Text Search.........................................................................................................395Using Greenplum MapReduce............................................................................................. 430Query Performance.............................................................................................................. 438Managing Spill Files Generated by Queries........................................................................ 438Query Profiling...................................................................................................................... 438

Working with External Data.............................................................................................................444Accessing External Data with PXF...................................................................................... 444Defining External Tables...................................................................................................... 444Accessing External Data with Foreign Tables..................................................................... 462Using the Greenplum Parallel File Server (gpfdist)..............................................................471

Loading and Unloading Data.......................................................................................................... 475Loading Data Using an External Table................................................................................ 476Loading and Writing Non-HDFS Custom Data.................................................................... 476Handling Load Errors............................................................................................................479Loading Data with gpload.....................................................................................................481Accessing External Data with PXF...................................................................................... 482Transforming External Data with gpfdist and gpload........................................................... 483Loading Data with COPY..................................................................................................... 493Running COPY in Single Row Error Isolation Mode............................................................494Optimizing Data Load and Query Performance................................................................... 494Unloading Data from Greenplum Database......................................................................... 494Formatting Data Files........................................................................................................... 497Example Custom Data Access Protocol.............................................................................. 500


6

Managing Performance................................................................................................................... 507Defining Database Performance.......................................................................................... 507Common Causes of Performance Issues............................................................................ 508Greenplum Database Memory Overview............................................................................. 511Managing Resources............................................................................................................515Investigating a Performance Problem.................................................................................. 544

Chapter 5: Greenplum Database Security Configuration Guide......... 547Securing the Database....................................................................................................................548Greenplum Database Ports and Protocols..................................................................................... 549Configuring Client Authentication.................................................................................................... 553

Allowing Connections to Greenplum Database....................................................................553Editing the pg_hba.conf File.................................................................................................555Authentication Methods........................................................................................................ 556SSL Client Authentication.....................................................................................................559PAM Based Authentication...................................................................................................561Radius Authentication...........................................................................................................562Limiting Concurrent Connections......................................................................................... 562Encrypting Client/Server Connections..................................................................................563

Configuring Database Authorization................................................................................................564Access Permissions and Roles............................................................................................564Managing Object Privileges..................................................................................................564Using SSH-256 Encryption...................................................................................................565Restricting Access by Time..................................................................................................567Dropping a Time-based Restriction.................................................................................... 569

Greenplum Command Center Security........................................................................................... 570Auditing............................................................................................................................................ 573Encrypting Data and Database Connections.................................................................................. 578

Encrypting gpfdist Connections............................................................................................ 578Encrypting Data at Rest with pgcrypto.................................................................................579

Security Best Practices................................................................................................................... 587

Chapter 6: Greenplum Database Best Practices..................................591Best Practices Summary.................................................................................................................592System Configuration...................................................................................................................... 598Schema Design............................................................................................................................... 603

Data Types........................................................................................................................... 603Storage Model...................................................................................................................... 603Compression......................................................................................................................... 604Distributions.......................................................................................................................... 605Partitioning............................................................................................................................ 608Indexes..................................................................................................................................610Column Sequence and Byte Alignment............................................................................... 610

Memory and Resource Management with Resource Groups......................................................... 612Memory and Resource Management with Resource Queues........................................................ 615System Monitoring and Maintenance..............................................................................................619

Monitoring............................................................................................................................. 619Updating Statistics with ANALYZE.......................................................................................620Managing Bloat in the Database..........................................................................................621Monitoring Greenplum Database Log Files..........................................................................625

Loading Data................................................................................................................................... 627INSERT Statement with Column Values..............................................................................627COPY Statement.................................................................................................................. 627External Tables.....................................................................................................................627


7

External Tables with Gpfdist................................................................................................ 627Gpload...................................................................................................................................628Best Practices.......................................................................................................................629

Security............................................................................................................................................ 630Encrypting Data and Database Connections.................................................................................. 633Tuning SQL Queries....................................................................................................................... 642

How to Generate Explain Plans........................................................................................... 642How to Read Explain Plans................................................................................................. 642Optimizing Greenplum Queries............................................................................................ 644

High Availability............................................................................................................................... 646Disk Storage......................................................................................................................... 646Master Mirroring....................................................................................................................646Segment Mirroring................................................................................................................ 647Dual Clusters........................................................................................................................ 648Backup and Restore.............................................................................................................648Detecting Failed Master and Segment Instances................................................................ 649Segment Mirroring Configuration..........................................................................................650

Chapter 7: Greenplum Database Utility Guide..................................... 655About the Greenplum Database Utilities.........................................................................................656

Referencing IP Addresses....................................................................................................656Running Backend Server Programs.....................................................................................656

Utility Reference.............................................................................................................................. 658analyzedb..............................................................................................................................658clusterdb................................................................................................................................662createdb................................................................................................................................ 664createlang............................................................................................................................. 666createuser............................................................................................................................. 667dropdb................................................................................................................................... 670droplang................................................................................................................................ 672dropuser................................................................................................................................ 673gpactivatestandby................................................................................................................. 674gpaddmirrors......................................................................................................................... 676gpbackup_manager.............................................................................................................. 679gpbackup...............................................................................................................................683gpcheckcat............................................................................................................................ 689gpcheckperf...........................................................................................................................692gpconfig.................................................................................................................................695gpcopy...................................................................................................................................698gpdeletesystem..................................................................................................................... 698gpexpand.............................................................................................................................. 699gpfdist....................................................................................................................................702gpinitstandby......................................................................................................................... 706gpinitsystem.......................................................................................................................... 708gpload................................................................................................................................... 715gplogfilter...............................................................................................................................726gpmapreduce........................................................................................................................ 728gpmapreduce.yaml................................................................................................................730gpmovemirrors...................................................................................................................... 737gpperfmon_install..................................................................................................................738gppkg.................................................................................................................................... 742gprecoverseg........................................................................................................................ 744gpreload................................................................................................................................ 748gprestore............................................................................................................................... 750gpscp.....................................................................................................................................755


8

gpssh.....................................................................................................................................757gpssh-exkeys........................................................................................................................ 760gpstart................................................................................................................................... 762gpstate.................................................................................................................................. 764gpstop................................................................................................................................... 768gpsys1...................................................................................................................................771pg_config...............................................................................................................................771pg_dump............................................................................................................................... 773pg_dumpall............................................................................................................................782pg_restore............................................................................................................................. 786pgbouncer............................................................................................................................. 791pgbouncer.ini.........................................................................................................................793pgbouncer-admin.................................................................................................................. 805psql........................................................................................................................................813reindexdb.............................................................................................................................. 838vacuumdb..............................................................................................................................840

Additional Supplied Programs......................................................................................................... 843

Chapter 8: Greenplum Database Reference Guide.............................. 844SQL Commands.............................................................................................................................. 845

SQL Syntax Summary..........................................................................................................848ABORT..................................................................................................................................886ALTER AGGREGATE...........................................................................................................887ALTER COLLATION.............................................................................................................888ALTER CONVERSION......................................................................................................... 889ALTER DATABASE.............................................................................................................. 890ALTER DEFAULT PRIVILEGES.......................................................................................... 891ALTER DOMAIN...................................................................................................................894ALTER EXTENSION.............................................................................................................896ALTER EXTERNAL TABLE..................................................................................................898ALTER FOREIGN DATA WRAPPER...................................................................................900ALTER FOREIGN TABLE.................................................................................................... 901ALTER FUNCTION...............................................................................................................905ALTER GROUP.................................................................................................................... 907ALTER INDEX...................................................................................................................... 908ALTER LANGUAGE............................................................................................................. 910ALTER MATERIALIZED VIEW.............................................................................................910ALTER OPERATOR............................................................................................................. 912ALTER OPERATOR CLASS................................................................................................ 913ALTER OPERATOR FAMILY...............................................................................................913ALTER PROTOCOL............................................................................................................. 916ALTER RESOURCE GROUP.............................................................................................. 917ALTER RESOURCE QUEUE...............................................................................................919ALTER ROLE....................................................................................................................... 922ALTER SCHEMA..................................................................................................................926ALTER SEQUENCE............................................................................................................. 927ALTER SERVER.................................................................................................................. 929ALTER TABLE......................................................................................................................930ALTER TABLESPACE..........................................................................................................943ALTER TEXT SEARCH CONFIGURATION.........................................................................944ALTER TEXT SEARCH DICTIONARY................................................................................ 946ALTER TEXT SEARCH PARSER........................................................................................947ALTER TEXT SEARCH TEMPLATE....................................................................................948ALTER TYPE........................................................................................................................948ALTER USER....................................................................................................................... 951


9

ALTER USER MAPPING..................................................................................................... 952ALTER VIEW........................................................................................................................ 953ANALYZE..............................................................................................................................954BEGIN................................................................................................................................... 958CHECKPOINT.......................................................................................................................960CLOSE.................................................................................................................................. 961CLUSTER............................................................................................................................. 961COMMENT............................................................................................................................963COMMIT................................................................................................................................966COPY.................................................................................................................................... 967CREATE AGGREGATE........................................................................................................979CREATE CAST.....................................................................................................................985CREATE COLLATION..........................................................................................................989CREATE CONVERSION...................................................................................................... 990CREATE DATABASE........................................................................................................... 991CREATE DOMAIN................................................................................................................993CREATE EXTENSION..........................................................................................................995CREATE EXTERNAL TABLE...............................................................................................996CREATE FOREIGN DATA WRAPPER..............................................................................1005CREATE FOREIGN TABLE............................................................................................... 1007CREATE FUNCTION..........................................................................................................1009CREATE GROUP............................................................................................................... 1017CREATE INDEX................................................................................................................. 1018CREATE LANGUAGE........................................................................................................ 1022CREATE MATERIALIZED VIEW........................................................................................1025CREATE OPERATOR........................................................................................................ 1026CREATE OPERATOR CLASS........................................................................................... 1031CREATE OPERATOR FAMILY..........................................................................................1034CREATE PROTOCOL........................................................................................................ 1035CREATE RESOURCE GROUP......................................................................................... 1036CREATE RESOURCE QUEUE..........................................................................................1039CREATE ROLE.................................................................................................................. 1043CREATE RULE...................................................................................................................1047CREATE SCHEMA.............................................................................................................1049CREATE SEQUENCE........................................................................................................ 1051CREATE SERVER............................................................................................................. 1053CREATE TABLE.................................................................................................................1055CREATE TABLE AS...........................................................................................................1068CREATE TABLESPACE.....................................................................................................1072CREATE TEXT SEARCH CONFIGURATION....................................................................1074CREATE TEXT SEARCH DICTIONARY............................................................................1075CREATE TEXT SEARCH PARSER...................................................................................1076CREATE TEXT SEARCH TEMPLATE...............................................................................1077CREATE TYPE...................................................................................................................1078CREATE USER.................................................................................................................. 1085CREATE USER MAPPING................................................................................................ 1086CREATE VIEW................................................................................................................... 1087DEALLOCATE.................................................................................................................... 1090DECLARE........................................................................................................................... 1090DELETE.............................................................................................................................. 1093DISCARD............................................................................................................................ 1096DO.......................................................................................................................................1097DROP AGGREGATE..........................................................................................................1098DROP CAST.......................................................................................................................1099DROP COLLATION............................................................................................................ 1100DROP CONVERSION........................................................................................................ 1101


10

DROP DATABASE............................................................................................................. 1102DROP DOMAIN.................................................................................................................. 1102DROP EXTENSION............................................................................................................1103DROP EXTERNAL TABLE.................................................................................................1104DROP FOREIGN DATA WRAPPER..................................................................................1105DROP FOREIGN TABLE................................................................................................... 1105DROP FUNCTION..............................................................................................................1106DROP GROUP................................................................................................................... 1107DROP INDEX..................................................................................................................... 1108DROP LANGUAGE.............................................................................................................1109DROP MATERIALIZED VIEW............................................................................................1109DROP OPERATOR............................................................................................................ 1110DROP OPERATOR CLASS............................................................................................... 1111DROP OPERATOR FAMILY.............................................................................................. 1112DROP OWNED...................................................................................................................1113DROP PROTOCOL............................................................................................................ 1114DROP RESOURCE GROUP..............................................................................................1114DROP RESOURCE QUEUE.............................................................................................. 1115DROP ROLE.......................................................................................................................1116DROP RULE.......................................................................................................................1117DROP SCHEMA................................................................................................................. 1118DROP SEQUENCE............................................................................................................ 1119DROP SERVER..................................................................................................................1119DROP TABLE..................................................................................................................... 1120DROP TABLESPACE.........................................................................................................1121DROP TEXT SEARCH CONFIGURATION........................................................................1122DROP TEXT SEARCH DICTIONARY................................................................................1123DROP TEXT SEARCH PARSER....................................................................................... 1124DROP TEXT SEARCH TEMPLATE...................................................................................1124DROP TYPE....................................................................................................................... 1125DROP USER...................................................................................................................... 1126DROP USER MAPPING.....................................................................................................1126DROP VIEW....................................................................................................................... 1127END.....................................................................................................................................1128EXECUTE........................................................................................................................... 1128EXPLAIN............................................................................................................................. 1129FETCH................................................................................................................................ 1134GRANT................................................................................................................................1137INSERT............................................................................................................................... 1142LOAD.................................................................................................................................. 1144LOCK.................................................................................................................................. 1145MOVE..................................................................................................................................1148PREPARE........................................................................................................................... 1150REASSIGN OWNED...........................................................................................................1151REFRESH MATERIALIZED VIEW..................................................................................... 1152REINDEX............................................................................................................................ 1154RELEASE SAVEPOINT......................................................................................................1155RESET................................................................................................................................ 1156REVOKE............................................................................................................................. 1157ROLLBACK......................................................................................................................... 1160ROLLBACK TO SAVEPOINT.............................................................................................1160SAVEPOINT........................................................................................................................1162SELECT.............................................................................................................................. 1163SELECT INTO.................................................................................................................... 1181SET..................................................................................................................................... 1182SET CONSTRAINTS.......................................................................................................... 1184


11

SET ROLE.......................................................................................................................... 1185SET SESSION AUTHORIZATION..................................................................................... 1186SET TRANSACTION.......................................................................................................... 1188SHOW................................................................................................................................. 1190START TRANSACTION..................................................................................................... 1191TRUNCATE.........................................................................................................................1193UPDATE..............................................................................................................................1194VACUUM.............................................................................................................................1198VALUES.............................................................................................................................. 1201

Data Types.................................................................................................................................... 1204Date/Time Types................................................................................................................ 1206Pseudo-Types..................................................................................................................... 1216Text Search Data Types.................................................................................................... 1218Range Types...................................................................................................................... 1220

Summary of Built-in Functions...................................................................................................... 1225Greenplum Database Function Types................................................................................1225Built-in Functions and Operators........................................................................................1226JSON Functions and Operators......................................................................................... 1229Window Functions.............................................................................................................. 1236Advanced Aggregate Functions......................................................................................... 1238Text Search Functions and Operators............................................................................... 1240Range Functions and Operators........................................................................................ 1244

Procedural Languages and Analytics Extensions.........................................................................1247PL/Container Language......................................................................................................1247PL/Java Language..............................................................................................................1271PL/Perl Language............................................................................................................... 1286PL/pgSQL Language.......................................................................................................... 1291PL/Python Language.......................................................................................................... 1298PL/R Language...................................................................................................................1307MADlib Extension for Analytics.......................................................................................... 1313PostGIS Geospatial Extension........................................................................................... 1323

Additional Supplied Modules......................................................................................................... 1330auto_explain........................................................................................................................1330citext....................................................................................................................................1330dblink...................................................................................................................................1331diskquota.............................................................................................................................1334fuzzystrmatch...................................................................................................................... 1338gp_sparse_vector................................................................................................................1338hstore.................................................................................................................................. 1342orafce.................................................................................................................................. 1343pageinspect.........................................................................................................................1345pgcrypto.............................................................................................................................. 1345sslinfo.................................................................................................................................. 1346

Character Set Support...................................................................................................................1347Setting the Character Set...................................................................................................1349Character Set Conversion Between Server and Client...................................................... 1349

Server Configuration Parameters..................................................................................................1352Parameter Types and Values.............................................................................................1352Setting Parameters............................................................................................................. 1352Parameter Categories.........................................................................................................1353Configuration Parameters...................................................................................................1363

System Catalogs........................................................................................................................... 1443System Tables.................................................................................................................... 1443System Views..................................................................................................................... 1444System Catalogs Definitions...............................................................................................1445

The gp_toolkit Administrative Schema..........................................................................................1545


12

Checking for Tables that Need Routine Maintenance........................................................1545Checking for Locks.............................................................................................................1546Checking Append-Optimized Tables.................................................................................. 1548Viewing Greenplum Database Server Log Files................................................................ 1552Checking Server Configuration Files..................................................................................1555Checking for Failed Segments........................................................................................... 1556Checking Resource Group Activity and Status.................................................................. 1557Checking Resource Queue Activity and Status................................................................. 1561Checking Query Disk Spill Space Usage...........................................................................1563Viewing Users and Groups (Roles)....................................................................................1565Checking Database Object Sizes and Disk Space............................................................ 1565Checking for Uneven Data Distribution.............................................................................. 1569Including Data for Materialized Views................................................................................ 1570

The gpperfmon Database..............................................................................................................1572database_*.........................................................................................................................1574diskspace_*....................................................................................................................... 1575interface_stats_*................................................................................................................ 1575log_alert_*..........................................................................................................................1577queries_*............................................................................................................................. 1578segment_*..........................................................................................................................1580socket_stats_*.....................................................................................................................1581system_*............................................................................................................................. 1582dynamic_memory_info........................................................................................................ 1584memory_info...................................................................................................................... 1584

Server Programmatic Interfaces....................................................................................................1586Greenplum Partner Connector API.................................................................................... 1586Developing a Background Worker Process....................................................................... 1605

SQL Features, Reserved and Key Words, and Compliance........................................................ 1608Summary of Greenplum Features...................................................................................... 1608Reserved Identifiers and SQL Key Words......................................................................... 1618SQL 2008 Optional Feature Compliance........................................................................... 1634

Chapter 10: Greenplum Client and Loader Tools Package............... 1664

Chapter 11: About the Tools Package................................................ 1665

Chapter 12: Installing the Client and Loader Tools Package............ 1666Installation Procedure....................................................................................................................1667About Your Installation.................................................................................................................. 1668Running the UNIX Tools Installer................................................................................................. 1669

Prerequisites....................................................................................................................... 1669Procedure............................................................................................................................1669

Running the Windows Tools Installer........................................................................................... 1670Prerequisites....................................................................................................................... 1670Procedure............................................................................................................................1670

Chapter 13: Configuring Greenplum Database for Remote ClientAccess................................................................................................. 1671


13

Chapter 14: Configuring a Client System for KerberosAuthentication.................................................................................... 1672

Chapter 15: Using the Client and Loader Tools................................. 1673Prerequisites.................................................................................................................................. 1674Setting Up Your Greenplum Database Clients Runtime Environment.......................................... 1675Running the Client and Loader Programs.................................................................................... 1676Greenplum Database Documentation References........................................................................1677Windows Considerations............................................................................................................... 1678

Chapter 16: Client and Loader Utility Reference................................1679

Chapter 17: DataDirect ODBC Drivers for Pivotal Greenplum...........1680Prerequisites.................................................................................................................................. 1681Supported Client Platforms........................................................................................................... 1682Installing on Linux Systems.......................................................................................................... 1683

Configuring the Driver on Linux......................................................................................... 1684Testing the Driver Connection on Linux.............................................................................1685

Installing on Windows Systems.................................................................................................... 1687Verifying the Version on Windows..................................................................................... 1687Configuring and Testing the Driver on Windows................................................................1687

DataDirect Driver Documentation..................................................................................................1689

Chapter 18: DataDirect JDBC Driver for Pivotal Greenplum............. 1690Prerequisites.................................................................................................................................. 1691Downloading the DataDirect JDBC Driver.................................................................................... 1692Obtaining Version Details for the Driver....................................................................................... 1693Usage Information......................................................................................................................... 1694DataDirect Driver Documentation..................................................................................................1695

Pivotal Greenplum 6.2 Release Notes Release Notes

14

Chapter 2

Pivotal Greenplum 6.2 Release Notes

This document contains pertinent release information about Pivotal Greenplum Database 6.2 releases.For previous versions of the release notes for Greenplum Database, go to Pivotal Greenplum DatabaseDocumentation. For information about Greenplum Database end of life, see Pivotal Greenplum Databaseend of life policy.

Pivotal Greenplum 6 software is available for download from the Pivotal Greenplum page on PivotalNetwork.

Pivotal Greenplum 6 is based on the open source Greenplum Database project code.

Important: Pivotal Support does not provide support for open source versions of GreenplumDatabase. Only Pivotal Greenplum Database is supported by Pivotal Support.

https://gpdb.docs.pivotal.io/https://gpdb.docs.pivotal.io/https://support.pivotal.io/hc/en-us/articles/201143553-GPDB-software-EOL-policyhttps://support.pivotal.io/hc/en-us/articles/201143553-GPDB-software-EOL-policyhttps://network.pivotal.io/products/pivotal-gpdbhttps://network.pivotal.io/products/pivotal-gpdbhttp://greenplum.org/


15

Release 6.2.1Release Date: 2019-12-12

Pivotal Greenplum 6.2.1 is a minor release that includes new features and resolves several issues.

New FeaturesGreenplum Database 6.2.1 includes these new features:

• Greenplum Database supports materialized views. Materialized views are similar to views. Amaterialized view enables you to save a frequently used or complex query, then access the queryresults in a SELECT statement as if they were a table. Materialized views persist the query results ina table-like form. Materialized view data cannot be directly updated. To refresh the materialized viewdata, use the REFRESH MATERIALIZED VIEW command. See Creating and Managing MaterializedViews.

Note: Known Issues and Limitations describes a limitation of materialized view support inGreenplum 6.2.1.

• The gpinitsystem utility supports the --ignore-warnings option. The option controls thevalue returned by gpinitsystem when warnings or an error occurs. If you specify this option,gpinitsystem returns 0 if warnings occurred during system initialization, and returns a non-zero valueif a fatal error occurs. If this option is not specified, gpinitsystem returns 1 if initialization completeswith warnings, and returns value of 2 or greater if a fatal error occurs.

• PXF version 5.10.0 is included, which introduces several new and changed features and bug fixes. SeePXF Version 5.10.0 below.

PXF Version 5.10.0PXF 5.10.0 includes the following new and changed features:

• PXF has improved its performance when reading a large number of files from HDFS or an object store.• PXF bundles newer tomcat and jackson libraries.• The PXF JDBC Connector now supports pushdown of OR and NOT logical filter operators when

specified in a JDBC named query or in an external table query filter condition.• PXF supports writing Avro-format data to Hadoop and object stores. Refer to Reading and Writing

HDFS Avro Data for more information about this feature.• PXF is now certified with Hadoop 2.x and 3.1.x and Hive Server 2.x and 3.1, and bundles new and

upgraded Hadoop libraries to support these versions.• PXF supports Kerberos authentication to Hive Server 2.x and 3.1.x.• PXF supports per-server user impersonation configuration.• PXF supports concurrent access to multiple Kerberized Hadoop clusters. In previous releases of

Greenplum Database, PXF supported accessing a single Hadoop cluster secured with Kerberos, andthis Hadoop cluster must have been configured as the default PXF server.

• PXF introduces a new template file, pxf-site.xml, to specify the Kerberos and impersonationproperty settings for a Hadoop or JDBC server configuration. Refer to About Kerberos and UserImpersonation Configuration (pxf-site.xml) for more information about this file.

• PXF now supports connecting to Hadoop with a configurable Hadoop user identity. PXF previouslysupported only proxy access to Hadoop via the gpadmin Greenplum user.

• PXF version 5.10.0 deprecates the following configuration properties.

Note: These property settings continue to work.

file:/home/yozie/github/gpdb/gpdb-doc/dita/temp/pdf/gp-docs-pivotal-gpdb/pxf/hdfs_avro.htmlfile:/home/yozie/github/gpdb/gpdb-doc/dita/temp/pdf/gp-docs-pivotal-gpdb/pxf/hdfs_avro.htmlfile:/home/yozie/github/gpdb/gpdb-doc/dita/temp/pdf/gp-docs-pivotal-gpdb/pxf/cfg_server.html#pxf-sitefile:/home/yozie/github/gpdb/gpdb-doc/dita/temp/pdf/gp-docs-pivotal-gpdb/pxf/cfg_server.html#pxf-site


16

• The PXF_USER_IMPERSONATION, PXF_PRINCIPAL, and PXF_KEYTAB settings in the pxf-env.sh file. You can use the pxf-site.xml file to configure Kerberos and impersonation settingsfor your new Hadoop server configurations.

• The pxf.impersonation.jdbc property setting in the jdbc-site.xml file. You can use thepxf.service.user.impersonation property to configure user impersonation for a new JDBCserver configuration.

Note: If you have previously configured a PXF JDBC server to access Kerberos-secured Hive, youmust upgrade the server definition. See Upgrading PXF in Greenplum 6.x for more information.

Changed FeaturesGreenplum Database 6.2.1 includes these changed features:

• Greenplum Stream Server version 1.3.1 is included in the Greenplum distribution.

Resolved IssuesPivotal Greenplum 6.2.1 is a minor release that resolves these issues:

29454 - gpstart

During Greenplum Database start up, the gpstart utility did not report when a segmentinstance failed to start. The utility always displayed 0 skipped segment starts. This issuehas been resolved. gpstart output was also enhanced to provide additional warningsand summary information about the number of skipped segments. For example:

[WARNING]:-****************************************************************************[WARNING]:-There are 1 segment(s) marked down in the database[WARNING]:-To recover from this current state, review usage of the gprecoverseg[WARNING]:-management utility which will recover failed segment instance databases.[WARNING]:-****************************************************************************

30248, 9022 - DLL

Greenplum Database might generate a PANIC when an index is created on a column ofan append-optimized, column-oriented table if the index definition contains a WHERE clausethat references multiple columns. This issue has been resolved.

7545 - Postgres Planner

The Postgres Planner might return incorrect results for queries that contain a subquery inan EXISTS clause if the subquery includes a LIMIT [ 0| ALL| NULL] clause or anOFFSET NULL clause. This issue has been resolved.


A query that used the Postgres planner could return incorrect results if it specified a volatilefunction in a LIMIT clause (for example, LIMIT (random() * 10)). This occurredbecause Greenplum evaluated the LIMIT clause separately on each segment instance toobtain a preliminary limit, before evaluating it once again as the query was dispatched. Theproblem was fixed by ensuring that a volatile functions in a LIMIT clause functions are notpushed to segment instances for evaluation.


Fixed a problem in the Postgres planner that could result in the error variable notfound in subplan target list. The issue applied to join queries where a tablecolumn had a user prescribed CAST applied to it while being both in the select list and in a

file:/home/yozie/github/gpdb/gpdb-doc/dita/temp/pdf/gp-docs-pivotal-gpdb/pxf/upgrade_pxf_6x.htmlhttps://github.com/greenplum-db/gpdb/issues/9022https://github.com/greenplum-db/gpdb/issues/7545https://github.com/greenplum-db/gpdb/issues/8509


17

join condition. At the same time, the column was also part of a motion operator in the queryplan.

30200 - Metrics Collector

Greenplum Database 6 stores tablespaces with non-default names as symlinks in the$MASTER_DATA_DIRECTORY/pg_tblspc directory and the metrics collector did notdetect these tablespaces. The metrics collector now follows the symlinks to find the namesof the tablespace directories and the data directories located in those tablespaces. Afterenabling the new metrics collector the tablespaces may not be visible in GreenplumCommand Center for up to four hours.

30203 - Query Optimizer

When updating a table's distribution column, Greenplum Database returned an error thatstates an UPDATE statement cannot update distribution columns if a btree index is definedon the distribution column and the UPDATE command contains an IN clause. The errorwas returned when Greenplum Database fell back to the Postgres planner to attempt theUPDATE operation. This issue has been resolved. Now GPORCA supports the specifiedtype of updates.

30206 - gpinitsystem

An example in the gpinitsystem help output used an invalid option for specifying theplacement of mirror segment instances in a spread configuration. The correct option is --mirror_mode=spread. This issue has been resolved.

30227 - Server

Greenplum Database with resource groups enabled might generate a PANIC when usingan extension with improper debug_query_string settings. The cause was a messagecontext issue and it has been resolved.

30256 - analyzedb

When executing some queries against partitioned tables, GPROCA would fail because ofmissing root partition statistics. This was caused by the analyzedb utility not updating theroot partition statistics when generating the partitioned table statistics. This issue has nowbeen resolved.

30292 - External Table

When Greenplum Database attempted to access data from an external table, a PANICwas generated when Greenplum Database could not resolve the host name that isspecified in the external table definition. This issue has been resolved. Now GreenplumDatabase returns an error in the specified situation.

168881383 - PXF

PXF fixed a regression in file and directory name pattern matching that affected the*:text:multi profiles and S3 Select. This issue has been resolved. PXF now correctlyhandles wildcards specified in the LOCATION data path.

169694492 - Query Optimizer

For a table that has a column that is defined with a btree index, GPORCA fell back to thePostgres planner for queries that use IN clause against the column or an OR of simplecomparisons on the column such as col = 5 OR col = 7. Now GPORCA attempts togenerate a query plan that uses the index.

169806983 - Greenplum Stream Server

In some cases, reading from Kafka using the default MINIMAL_INTERVAL (0 seconds)caused GPSS to consume a large amount of CPU resources, even when no newmessages existed in the Kafka topic. This issue is resolved in GPSS 1.3.1.

169807372, 169831558 - Greenplum Stream Server


18

GPSS 1.3.0 did not recognize internal history tables that were created with GPSS 1.2.6and earlier. In some cases, this caused GPSS to load duplicate messages into GreenplumDatabase. This issue is resolved in GPSS 1.3.1.

170041280 - PXF

PXF was unable to read data from an encrypted HDFS zone and returned anorg.apache.hadoop.crypto.CryptoInputStream cannot be cast toorg.apache.hadoop.hdfs.DFSInputStream error in this situation. This issue isresolved.

Upgrading to Greenplum 6.2.1Note: Greenplum 6 does not support direct upgrades from Greenplum 4 or Greenplum 5 releases,or from earlier Greenplum 6 Beta releases.

See Upgrading from an Earlier Greenplum 6 Release to upgrade your existing Greenplum 6.x software toGreenplum 6.2.1.


19

Deprecated FeaturesDeprecated features will be removed in a future major release of Greenplum Database. Pivotal Greenplum6.x deprecates:

• The analzyedb option --skip_root_stats (deprecated since 6.2).

If the option is specified, a warning is issued stating that the option will be ignored.• The server configuration parameter gp_statistics_use_fkeys (deprecated since 6.2).• The following PXF configuration properties (deprecated since 6.2):

• The PXF_USER_IMPERSONATION, PXF_PRINCIPAL, and PXF_KEYTAB settings in the pxf-env.sh file. You can use the pxf-site.xml file to configure Kerberos and impersonation settingsfor your new Hadoop server configurations.

• The pxf.impersonation.jdbc property setting in the jdbc-site.xml file. You can use thepxf.service.user.impersonation property to configure user impersonation for a new JDBCserver configuration.

• The server configuration parameter gp_ignore_error_table (deprecated since 6.0).

To avoid a Greenplum Database syntax error, set the value of this parameter to true when yourun applications that execute CREATE EXTERNAL TABLE or COPY commands that include the nowremoved Greenplum Database 4.3.x INTO ERROR TABLE clause.

• Specifying => as an operator name in the CREATE OPERATOR command (deprecated since 6.0).• The Greenplum external table C API (deprecated since 6.0).

Any developers using this API are encouraged to use the new Foreign Data Wrapper API in its place.• Commas placed between a SUBPARTITION TEMPLATE clause and its corresponding SUBPARTITION

BY clause, and between consecutive SUBPARTITION BY clauses in a CREATE TABLE command(deprecated since 6.0).

Using this undocumented syntax will generate a deprecation warning message.• The timestamp format YYYYMMDDHH24MISS (deprecated since 6.0).

This format could not be parsed unambiguously in previous Greenplum Database releases, and is notsupported in PostgreSQL 9.4.

• The createlang and droplang utilities (deprecated since 6.0).• The pg_resqueue_status system view (deprecated since 6.0).

Use the gp_toolkit.gp_resqueue_status view instead.• The GLOBAL and LOCAL modifiers when creating a temporary table with the CREATE TABLE and

CREATE TABLE AS commands (deprecated since 6.0).

These keywords are present for SQL standard compatibility, but have no effect in Greenplum Database.• The Greenplum Platform Extension Framework (PXF) HDFS profile names for the Text, Avro, JSON,

Parquet, and SequenceFile data formats (deprecated since 5.16).

Refer to Connectors, Data Formats, and Profiles in the PXF Hadoop documentation for moreinformation.

• Using WITH OIDS or oids=TRUE to assign an OID system column when creating or altering a table(deprecated since 6.0).

• Allowing superusers to specify the SQL_ASCII encoding regardless of the locale settings (deprecatedsince 6.0).

This choice may result in misbehavior of character-string functions when data that is not encoding-compatible with the locale is stored in the database.

• The @@@ text search operator (deprecated since 6.0).

This operator is currently a synonym for the @@ operator.

../pxf/access_hdfs.html#hadoop_connectors


20

• The unparenthesized syntax for option lists in the VACUUM command (deprecated since 6.0).

This syntax requires that the options to the command be specified in a specific order.• The plain pgbouncer authentication type (auth_type = plain) (deprecated since 4.x).


21

Migrating Data to Greenplum 6Note: Greenplum 6 does not support direct upgrades from Greenplum 4 or Greenplum 5 releases,or from earlier Greenplum 6 Beta releases.

See Migrating Data from Greenplum 4.3 or 5 for guidelines and considerations for migrating existingGreenplum data to Greenplum 6, using standard backup and restore procedures.


22

Known Issues and LimitationsPivotal Greenplum 6 has these limitations:

• Upgrading a Greenplum Database 4 or 5 release, or Greenplum 6 Beta release, to Pivotal Greenplum 6is not supported.

• MADlib, GPText, and PostGIS are not yet provided for installation on Ubuntu systems.• Greenplum 6 is not supported for installation on DCA systems.• Greenplum for Kubernetes is not yet provided with this release.

The following table lists key known issues in Pivotal Greenplum 6.x.

Table 1: Key Known Issues in Pivotal Greenplum 6.x

Issue Category Description

n/a MaterializedViews

By default, certain gp_toolkit views do not display data formaterialized views. If you want to include this information in gp_toolkit view output, you must redefine a gp_toolkit internalview as described in Including Data for Materialized Views.

168689202 PXF PXF fails to run any query on Java 11 that specifies a Hive

Documents

Pivotal Greenplum Database · Contents Release Notes 6 Managing Performance..... 507