Pivotal Greenplum Database · 2020. 7. 1. · Chapter 2: Pivotal Greenplum 5.10.1 Release Notes ... ETL Hosts for Data Loading ... 536 Loading Data with gpload.....538 Transforming

PRODUCT DOCUMENTATION

Pivotal™ GreenplumDatabase®Version 5.10.2

Pivotal Greenplum DatabaseDocumentationRev: A01

© 2018 Pivotal Software, Inc.

Copyright OpenTopic

2

Notice

Copyright

Privacy Policy | Terms of Use

Copyright © 2018 Pivotal Software, Inc. All rights reserved.

Pivotal Software, Inc. believes the information in this publication is accurate as of its publication date. Theinformation is subject to change without notice. THE INFORMATION IN THIS PUBLICATION IS PROVIDED"AS IS." PIVOTAL SOFTWARE, INC. ("Pivotal") MAKES NO REPRESENTATIONS OR WARRANTIES OF ANYKIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMSIMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Use, copying, and distribution of any Pivotal software described in this publication requires an applicablesoftware license.

All trademarks used herein are the property of Pivotal or their respective owners.

Revised August 2018 (5.10.2)

http://pivotal.io/privacy-policyhttp://pivotal.io/terms-of-use

Contents OpenTopic

3

Contents

Chapter 1: Pivotal Greenplum 5.10.2 Release Notes............................. 15Welcome to Pivotal Greenplum 5.10.2............................................................................................. 16Experimental Features...................................................................................................................... 17Differences Compared to Open Source Greenplum Database.........................................................18Supported Platforms..........................................................................................................................19

Veritas NetBackup.................................................................................................................. 21Supported Platform Notes...................................................................................................... 21

Pivotal Greenplum Tools and Extensions Compatibility................................................................... 23Client Tools.............................................................................................................................23Extensions...............................................................................................................................24Pivotal Greenplum Data Connectors......................................................................................25Pivotal GPText Compatibility.................................................................................................. 25Pivotal Greenplum Command Center.................................................................................... 25

Hadoop Distribution Compatibility..................................................................................................... 26Upgrading to Greenplum Database 5.10.2....................................................................................... 27

Upgrading from 5.x to 5.10.2................................................................................................. 28Troubleshooting a Failed Upgrade.........................................................................................29

Migrating Data to Pivotal Greenplum 5.x..........................................................................................30Pivotal Greenplum on DCA Systems................................................................................................32

Installing the Pivotal Greenplum 5.10.2 Software Binaries on DCA Systems........................ 32Upgrading from 5.x to 5.10.2 on DCA Systems.....................................................................32

Resolved Issues................................................................................................................................ 34Known Issues and Limitations.......................................................................................................... 35Update for gp_toolkit.gp_bloat_diag Issue........................................................................................43

Chapter 2: Pivotal Greenplum 5.10.1 Release Notes............................. 45Welcome to Pivotal Greenplum 5.10.1............................................................................................. 46Experimental Features...................................................................................................................... 47Differences Compared to Open Source Greenplum Database.........................................................48Supported Platforms..........................................................................................................................49







Resolved Issues................................................................................................................................ 64Known Issues and Limitations.......................................................................................................... 65

Contents OpenTopic

4

Update for gp_toolkit.gp_bloat_diag Issue........................................................................................73

Chapter 3: Pivotal Greenplum 5.10.0 Release Notes............................. 75Welcome to Pivotal Greenplum 5.10.0............................................................................................. 76New Features.................................................................................................................................... 77

gpcopy Enhancements........................................................................................................... 77Bypass Resource Group Concurrent Transaction Limits....................................................... 77gpload Performance Enhancement........................................................................................ 78gpbackup S3 Plugin Enhancements...................................................................................... 78Storage Plugin API Execution Scope.....................................................................................78Filter Pushdown for External Table Protocols........................................................................79Pivotal Greenplum-Kafka Connector (Experimental)..............................................................79

Changed Features.............................................................................................................................80Experimental Features...................................................................................................................... 81Differences Compared to Open Source Greenplum Database.........................................................82Supported Platforms..........................................................................................................................83







Resolved Issues................................................................................................................................ 98Known Issues and Limitations........................................................................................................ 102Update for gp_toolkit.gp_bloat_diag Issue......................................................................................110

Chapter 4: Greenplum Database Installation Guide.............................112Introduction to Greenplum...............................................................................................................113

The Greenplum Master.........................................................................................................113The Segments...................................................................................................................... 114The Interconnect...................................................................................................................115ETL Hosts for Data Loading................................................................................................ 116Greenplum Performance Monitoring.................................................................................... 117

Estimating Storage Capacity........................................................................................................... 118Calculating Usable Disk Capacity........................................................................................ 118Calculating User Data Size.................................................................................................. 118Calculating Space Requirements for Metadata and Logs....................................................119

Configuring Your Systems and Installing Greenplum..................................................................... 120System Requirements...........................................................................................................120Setting the Greenplum Recommended OS Parameters...................................................... 122Creating the Greenplum Database Administrative User Account........................................ 126Installing the Greenplum Database Software.......................................................................127Installing and Configuring Greenplum on all Hosts..............................................................128Installing Oracle Compatibility Functions............................................................................. 130

Contents OpenTopic

5

Installing Optional Modules.................................................................................................. 130Installing Greenplum Database Extensions..........................................................................131Installing and Configuring the Greenplum Platform Extension Framework (PXF)................ 131Creating the Data Storage Areas.........................................................................................131Synchronizing System Clocks.............................................................................................. 133Enabling iptables.................................................................................................................. 133Amazon EC2 Configuration (Amazon Web Services)..........................................................136Next Steps............................................................................................................................ 141

Installing the Data Science Packages............................................................................................ 142Python Data Science Module Package................................................................................142R Data Science Library Package......................................................................................... 144

Validating Your Systems................................................................................................................. 147Validating OS Settings..........................................................................................................147Validating Hardware Performance........................................................................................147Validating Disk I/O and Memory Bandwidth.........................................................................148

Initializing a Greenplum Database System..................................................................................... 150Overview............................................................................................................................... 150Initializing Greenplum Database...........................................................................................150Setting Greenplum Environment Variables.......................................................................... 153Next Steps............................................................................................................................ 154

Configuring Timezone and Localization Settings............................................................................156Configuring the Timezone.................................................................................................... 156About Locale Support in Greenplum Database................................................................... 156Character Set Support..........................................................................................................158Setting the Character Set.....................................................................................................160Character Set Conversion Between Server and Client........................................................161

About Implicit Text Casting in Greenplum Database......................................................................164Workaround: Manually Creating Missing Operators.............................................................165

Installation Management Utilities.....................................................................................................168Greenplum Environment Variables................................................................................................. 169

Required Environment Variables..........................................................................................169Optional Environment Variables........................................................................................... 169

Chapter 5: Greenplum Database Administrator Guide........................ 171Greenplum Database Concepts...................................................................................................... 172

About the Greenplum Architecture....................................................................................... 172About Management and Monitoring Utilities........................................................................ 174About Concurrency Control in Greenplum Database...........................................................175About Parallel Data Loading................................................................................................ 183About Redundancy and Failover in Greenplum Database...................................................184About Database Statistics in Greenplum Database............................................................. 186

Managing a Greenplum System..................................................................................................... 193About the Greenplum Database Release Version Number................................................. 193Starting and Stopping Greenplum Database....................................................................... 193Accessing the Database.......................................................................................................196Configuring the Greenplum Database System.....................................................................204Enabling High Availability and Data Consistency Features................................................. 215Backing Up and Restoring Databases................................................................................. 233Expanding a Greenplum System..........................................................................................287Migrating Data...................................................................................................................... 302Monitoring a Greenplum System..........................................................................................313Routine System Maintenance Tasks....................................................................................333Recommended Monitoring and Maintenance Tasks............................................................ 337

Managing Greenplum Database Access.........................................................................................345Configuring Client Authentication......................................................................................... 345

Contents OpenTopic

6

Managing Roles and Privileges............................................................................................371Defining Database Objects..............................................................................................................378

Creating and Managing Databases......................................................................................378Creating and Managing Tablespaces...................................................................................380Creating and Managing Schemas........................................................................................383Creating and Managing Tables............................................................................................ 384Choosing the Table Storage Model..................................................................................... 387Partitioning Large Tables......................................................................................................398Creating and Using Sequences........................................................................................... 410Using Indexes in Greenplum Database............................................................................... 413Creating and Managing Views............................................................................................. 416

Distribution and Skew..................................................................................................................... 417Local (Co-located) Joins.......................................................................................................417Data Skew............................................................................................................................ 417Processing Skew.................................................................................................................. 418

Inserting, Updating, and Deleting Data...........................................................................................421About Concurrency Control in Greenplum Database...........................................................421Inserting Rows...................................................................................................................... 422Updating Existing Rows........................................................................................................423Deleting Rows.......................................................................................................................423Working With Transactions...................................................................................................423Vacuuming the Database..................................................................................................... 425

Querying Data................................................................................................................................. 426About Greenplum Query Processing....................................................................................426About GPORCA....................................................................................................................429Defining Queries................................................................................................................... 442WITH Queries (Common Table Expressions)......................................................................452Using Functions and Operators............................................................................................455Working with JSON Data..................................................................................................... 464Working with XML Data........................................................................................................468Query Performance.............................................................................................................. 480Managing Spill Files Generated by Queries........................................................................ 480Query Profiling...................................................................................................................... 480

Working with External Data.............................................................................................................486Defining External Tables...................................................................................................... 486Accessing External Data with PXF...................................................................................... 504Accessing HDFS Data with gphdfs...................................................................................... 505Using the Greenplum Parallel File Server (gpfdist)..............................................................528

Loading and Unloading Data.......................................................................................................... 532Loading Data Using an External Table................................................................................ 533Loading and Writing Non-HDFS Custom Data.................................................................... 533Handling Load Errors............................................................................................................536Loading Data with gpload.....................................................................................................538Transforming External Data with gpfdist and gpload........................................................... 539Loading Data with COPY..................................................................................................... 549Running COPY in Single Row Error Isolation Mode............................................................549Optimizing Data Load and Query Performance................................................................... 549Unloading Data from Greenplum Database......................................................................... 550Formatting Data Files........................................................................................................... 552Example Custom Data Access Protocol.............................................................................. 555

Managing Performance................................................................................................................... 562Defining Database Performance.......................................................................................... 562Common Causes of Performance Issues............................................................................ 563Greenplum Database Memory Overview............................................................................. 566Managing Resources............................................................................................................569Investigating a Performance Problem.................................................................................. 597

Contents OpenTopic

7

Chapter 6: Greenplum Database Security Configuration Guide......... 600Securing the Database....................................................................................................................601Greenplum Database Ports and Protocols..................................................................................... 602Configuring Client Authentication.................................................................................................... 606

Allowing Connections to Greenplum Database....................................................................606Editing the pg_hba.conf File.................................................................................................607Authentication Methods........................................................................................................ 608SSL Client Authentication.....................................................................................................611PAM Based Authentication...................................................................................................613Radius Authentication...........................................................................................................613Limiting Concurrent Connections......................................................................................... 613Encrypting Client/Server Connections..................................................................................614

Configuring Database Authorization................................................................................................616Access Permissions and Roles............................................................................................616Managing Object Privileges..................................................................................................616Using SSH-256 Encryption...................................................................................................617Restricting Access by Time..................................................................................................619Dropping a Time-based Restriction.................................................................................... 621

Greenplum Command Center Security........................................................................................... 622Auditing............................................................................................................................................ 625Encrypting Data and Database Connections.................................................................................. 630

Encrypting gpfdist Connections............................................................................................ 630Encrypting Data at Rest with pgcrypto.................................................................................631

Enabling gphdfs Authentication with a Kerberos-secured Hadoop Cluster.....................................639Prerequisites......................................................................................................................... 639Configuring the Greenplum Cluster......................................................................................639Creating and Installing Keytab Files.................................................................................... 640Configuring gphdfs for Kerberos.......................................................................................... 641Testing Greenplum Database Access to HDFS...................................................................642Troubleshooting HDFS with Kerberos..................................................................................643

Security Best Practices................................................................................................................... 645

Chapter 7: Greenplum Database Best Practices..................................649Best Practices Summary.................................................................................................................650System Configuration...................................................................................................................... 656Schema Design............................................................................................................................... 661

Data Types........................................................................................................................... 661Storage Model...................................................................................................................... 661Compression......................................................................................................................... 662Distributions.......................................................................................................................... 663Partitioning............................................................................................................................ 666Indexes..................................................................................................................................668Column Sequence and Byte Alignment............................................................................... 668

Memory and Resource Management with Resource Queues........................................................ 670System Monitoring and Maintenance..............................................................................................674

Monitoring............................................................................................................................. 674Updating Statistics with ANALYZE.......................................................................................675Managing Bloat in the Database..........................................................................................676Monitoring Greenplum Database Log Files..........................................................................680

Loading Data................................................................................................................................... 682INSERT Statement with Column Values..............................................................................682COPY Statement.................................................................................................................. 682External Tables.....................................................................................................................682

Contents OpenTopic

8

External Tables with Gpfdist................................................................................................ 682Gpload...................................................................................................................................683Best Practices.......................................................................................................................684

Migrating Data with gptransfer........................................................................................................ 685Security............................................................................................................................................ 691Encrypting Data and Database Connections.................................................................................. 694Tuning SQL Queries....................................................................................................................... 703

How to Generate Explain Plans........................................................................................... 703How to Read Explain Plans................................................................................................. 703Optimizing Greenplum Queries............................................................................................ 705

High Availability............................................................................................................................... 707Disk Storage......................................................................................................................... 707Master Mirroring....................................................................................................................707Segment Mirroring................................................................................................................ 708Dual Clusters........................................................................................................................ 709Backup and Restore.............................................................................................................709Detecting Failed Master and Segment Instances................................................................ 710Segment Mirroring Configuration..........................................................................................711

Chapter 8: Greenplum Database Utility Guide..................................... 717Management Utility Reference........................................................................................................ 718

Backend Server Programs................................................................................................... 719analyzedb..............................................................................................................................720gpactivatestandby................................................................................................................. 724gpaddmirrors......................................................................................................................... 726gpbackup...............................................................................................................................729gpcheck.................................................................................................................................734gpcheckcat............................................................................................................................ 736gpcheckperf...........................................................................................................................738gpconfig.................................................................................................................................741gpcrondump.......................................................................................................................... 745gpdbrestore........................................................................................................................... 758gpcopy...................................................................................................................................766gpdeletesystem..................................................................................................................... 774gpexpand.............................................................................................................................. 775gpfdist....................................................................................................................................778gpfilespace............................................................................................................................ 782gpinitstandby......................................................................................................................... 785gpinitsystem.......................................................................................................................... 787gpload................................................................................................................................... 794gplogfilter...............................................................................................................................804gpmapreduce........................................................................................................................ 807gpmfr..................................................................................................................................... 808gpmovemirrors...................................................................................................................... 812gpperfmon_install..................................................................................................................813gppkg.................................................................................................................................... 817gprecoverseg........................................................................................................................ 819gpreload................................................................................................................................ 824gprestore............................................................................................................................... 826gpscp.....................................................................................................................................830gpseginstall........................................................................................................................... 832gpssh.....................................................................................................................................834gpssh-exkeys........................................................................................................................ 836gpstart................................................................................................................................... 839gpstate.................................................................................................................................. 841

Contents OpenTopic

9

gpstop................................................................................................................................... 844gpsys1...................................................................................................................................847gptransfer.............................................................................................................................. 848pgbouncer............................................................................................................................. 860pgbouncer.ini.........................................................................................................................861pgbouncer-admin.................................................................................................................. 873

Client Utility Reference....................................................................................................................883Client Utility Summary.......................................................................................................... 883

Additional Supplied Modules........................................................................................................... 933citext Data Type................................................................................................................... 933dblink Functions....................................................................................................................935hstore Functions................................................................................................................... 936Oracle Compatibility Functions.............................................................................................939passwordcheck..................................................................................................................... 960

Chapter 9: Greenplum Database Reference Guide.............................. 962SQL Command Reference..............................................................................................................963

SQL Syntax Summary..........................................................................................................965ABORT..................................................................................................................................992ALTER AGGREGATE...........................................................................................................993ALTER CONVERSION......................................................................................................... 994ALTER DATABASE.............................................................................................................. 995ALTER DOMAIN...................................................................................................................996ALTER EXTENSION.............................................................................................................998ALTER EXTERNAL TABLE................................................................................................1000ALTER FILESPACE............................................................................................................1002ALTER FUNCTION.............................................................................................................1003ALTER GROUP.................................................................................................................. 1005ALTER INDEX.................................................................................................................... 1006ALTER LANGUAGE........................................................................................................... 1007ALTER OPERATOR........................................................................................................... 1008ALTER OPERATOR CLASS.............................................................................................. 1009ALTER OPERATOR FAMILY.............................................................................................1009ALTER PROTOCOL........................................................................................................... 1012ALTER RESOURCE GROUP............................................................................................ 1013ALTER RESOURCE QUEUE.............................................................................................1015ALTER ROLE..................................................................................................................... 1018ALTER SCHEMA................................................................................................................1021ALTER SEQUENCE........................................................................................................... 1022ALTER TABLE....................................................................................................................1024ALTER TABLESPACE........................................................................................................1034ALTER TYPE......................................................................................................................1035ALTER USER..................................................................................................................... 1036ALTER VIEW...................................................................................................................... 1036ANALYZE............................................................................................................................1037BEGIN................................................................................................................................. 1040CHECKPOINT.....................................................................................................................1042CLOSE................................................................................................................................ 1042CLUSTER........................................................................................................................... 1043COMMENT..........................................................................................................................1044COMMIT..............................................................................................................................1046COPY.................................................................................................................................. 1047CREATE AGGREGATE......................................................................................................1057CREATE CAST...................................................................................................................1061CREATE CONVERSION.................................................................................................... 1064

Contents OpenTopic

10

CREATE DATABASE......................................................................................................... 1065CREATE DOMAIN..............................................................................................................1066CREATE EXTENSION........................................................................................................1068CREATE EXTERNAL TABLE.............................................................................................1069CREATE FUNCTION..........................................................................................................1078CREATE GROUP............................................................................................................... 1084CREATE INDEX................................................................................................................. 1085CREATE LANGUAGE........................................................................................................ 1088CREATE OPERATOR........................................................................................................ 1090CREATE OPERATOR CLASS........................................................................................... 1094CREATE OPERATOR FAMILY..........................................................................................1097CREATE PROTOCOL........................................................................................................ 1098CREATE RESOURCE GROUP......................................................................................... 1099CREATE RESOURCE QUEUE..........................................................................................1102CREATE ROLE.................................................................................................................. 1106CREATE RULE...................................................................................................................1110CREATE SCHEMA.............................................................................................................1112CREATE SEQUENCE........................................................................................................ 1113CREATE TABLE.................................................................................................................1116CREATE TABLE AS...........................................................................................................1127CREATE TABLESPACE.....................................................................................................1130CREATE TYPE...................................................................................................................1131CREATE USER.................................................................................................................. 1137CREATE VIEW................................................................................................................... 1138DEALLOCATE.................................................................................................................... 1139DECLARE........................................................................................................................... 1140DELETE.............................................................................................................................. 1142DISCARD............................................................................................................................ 1144DO.......................................................................................................................................1145DROP AGGREGATE..........................................................................................................1147DROP CAST.......................................................................................................................1148DROP CONVERSION........................................................................................................ 1148DROP DATABASE............................................................................................................. 1149DROP DOMAIN.................................................................................................................. 1150DROP EXTENSION............................................................................................................1151DROP EXTERNAL TABLE.................................................................................................1151DROP FILESPACE.............................................................................................................1152DROP FUNCTION..............................................................................................................1153DROP GROUP................................................................................................................... 1154DROP INDEX..................................................................................................................... 1154DROP LANGUAGE.............................................................................................................1155DROP OPERATOR............................................................................................................ 1156DROP OPERATOR CLASS............................................................................................... 1157DROP OPERATOR FAMILY.............................................................................................. 1157DROP OWNED...................................................................................................................1158DROP PROTOCOL............................................................................................................ 1159DROP RESOURCE GROUP..............................................................................................1160DROP RESOURCE QUEUE.............................................................................................. 1161DROP ROLE.......................................................................................................................1162DROP RULE.......................................................................................................................1163DROP SCHEMA................................................................................................................. 1164DROP SEQUENCE............................................................................................................ 1164DROP TABLE..................................................................................................................... 1165DROP TABLESPACE.........................................................................................................1166DROP TYPE....................................................................................................................... 1166DROP USER...................................................................................................................... 1167

Contents OpenTopic

11

DROP VIEW....................................................................................................................... 1168END.....................................................................................................................................1168EXECUTE........................................................................................................................... 1169EXPLAIN............................................................................................................................. 1170FETCH................................................................................................................................ 1172GRANT................................................................................................................................1175INSERT............................................................................................................................... 1179LOAD.................................................................................................................................. 1181LOCK.................................................................................................................................. 1181MOVE..................................................................................................................................1184PREPARE........................................................................................................................... 1185REASSIGN OWNED...........................................................................................................1187REINDEX............................................................................................................................ 1188RELEASE SAVEPOINT......................................................................................................1189RESET................................................................................................................................ 1190REVOKE............................................................................................................................. 1191ROLLBACK......................................................................................................................... 1193ROLLBACK TO SAVEPOINT.............................................................................................1193SAVEPOINT........................................................................................................................1194SELECT.............................................................................................................................. 1196SELECT INTO.................................................................................................................... 1210SET..................................................................................................................................... 1211SET ROLE.......................................................................................................................... 1213SET SESSION AUTHORIZATION..................................................................................... 1214SET TRANSACTION.......................................................................................................... 1215SHOW................................................................................................................................. 1217START TRANSACTION..................................................................................................... 1218TRUNCATE.........................................................................................................................1219UPDATE..............................................................................................................................1220VACUUM.............................................................................................................................1223VALUES.............................................................................................................................. 1226

SQL 2008 Optional Feature Compliance......................................................................................1228Greenplum Environment Variables............................................................................................... 1257

Required Environment Variables........................................................................................1257Optional Environment Variables......................................................................................... 1257

System Catalog Reference........................................................................................................... 1259System Tables.................................................................................................................... 1259System Views..................................................................................................................... 1260System Catalogs Definitions...............................................................................................1261

The gp_toolkit Administrative Schema..........................................................................................1346Checking for Tables that Need Routine Maintenance........................................................1346Checking for Locks.............................................................................................................1347Checking Append-Optimized Tables.................................................................................. 1349Viewing Greenplum Database Server Log Files................................................................ 1353Checking Server Configuration Files..................................................................................1356Checking for Failed Segments........................................................................................... 1357Checking Resource Group Activity and Status.................................................................. 1358Checking Resource Queue Activity and Status................................................................. 1360Checking Query Disk Spill Space Usage...........................................................................1362Viewing Users and Groups (Roles)....................................................................................1364Checking Database Object Sizes and Disk Space............................................................ 1365Checking for Uneven Data Distribution.............................................................................. 1369

The gpperfmon Database..............................................................................................................1370database_*.........................................................................................................................1372diskspace_*....................................................................................................................... 1373interface_stats_*................................................................................................................ 1373

Contents OpenTopic

12

log_alert_*..........................................................................................................................1375queries_*............................................................................................................................. 1376segment_*..........................................................................................................................1378socket_stats_*.....................................................................................................................1379system_*............................................................................................................................. 1380dynamic_memory_info........................................................................................................ 1382memory_info...................................................................................................................... 1382

Greenplum Database Data Types.................................................................................................1384Character Set Support...................................................................................................................1388

Setting the Character Set...................................................................................................1390Character Set Conversion Between Server and Client...................................................... 1390

Server Configuration Parameters..................................................................................................1393Parameter Types and Values.............................................................................................1393Setting Parameters............................................................................................................. 1393Parameter Categories.........................................................................................................1394Configuration Parameters...................................................................................................1404

Summary of Built-in Functions...................................................................................................... 1486Greenplum Database Function Types................................................................................1486Built-in Functions and Operators........................................................................................1487JSON Functions and Operators......................................................................................... 1490Window Functions.............................................................................................................. 1493Advanced Aggregate Functions......................................................................................... 1495

Greenplum MapReduce Specification...........................................................................................1497Greenplum MapReduce Document Format........................................................................1497Greenplum MapReduce Document Schema......................................................................1498Example Greenplum MapReduce Document..................................................................... 1505

Greenplum PL/pgSQL Procedural Language............................................................................... 1511About Greenplum Database PL/pgSQL............................................................................. 1511PL/pgSQL Plan Caching.....................................................................................................1513PL/pgSQL Examples...........................................................................................................1513References..........................................................................................................................1517

Greenplum PostGIS Extension..................................................................................................... 1518About PostGIS.................................................................................................................... 1518Enabling and Removing PostGIS Support......................................................................... 1519Usage..................................................................................................................................1520PostGIS Extension Support and Limitations...................................................................... 1521PostGIS Support Scripts.....................................................................................................1522

Greenplum PL/R Language Extension..........................................................................................1525About Greenplum Database PL/R......................................................................................1525

Greenplum PL/Python Language Extension................................................................................. 1531About Greenplum PL/Python..............................................................................................1531Enabling and Removing PL/Python support...................................................................... 1531Developing Functions with PL/Python................................................................................1532Installing Python Modules...................................................................................................1535Examples............................................................................................................................ 1538References..........................................................................................................................1540

Greenplum PL/Container Language Extension.............................................................................1541About the PL/Container Language Extension.................................................................... 1541About PL/Container Resource Management......................................................................1542PL/Container Docker Images............................................................................................. 1544Prerequisites....................................................................................................................... 1544Installing the PL/Container Language Extension............................................................... 1545Installing PL/Container Docker Images.............................................................................. 1548Uninstalling PL/Container................................................................................................... 1548Using PL/Container Functions............................................................................................1550About PL/Container Running PL/Python............................................................................ 1552

Contents OpenTopic

13

About PL/Container Running PL/R.....................................................................................1553Configuring PL/Container....................................................................................................1553Installing Docker................................................................................................................. 1563References..........................................................................................................................1564

Greenplum PL/Java Language Extension.....................................................................................1565About PL/Java.................................................................................................................... 1565About Greenplum Database PL/Java.................................................................................1566Installing PL/Java................................................................................................................1567Uninstalling PL/Java........................................................................................................... 1568Enabling PL/Java and Installing JAR Files........................................................................ 1569Writing PL/Java functions................................................................................................... 1569Using JDBC........................................................................................................................ 1575Exception Handling.............................................................................................................1575Savepoints.......................................................................................................................... 1575Logging............................................................................................................................... 1576Security............................................................................................................................... 1576Some PL/Java Issues and Solutions..................................................................................1577Example.............................................................................................................................. 1578References..........................................................................................................................1579

Greenplum PL/Perl Language Extension......................................................................................1580About Greenplum PL/Perl...................................................................................................1580Greenplum Database PL/Perl Limitations.......................................................................... 1580Trusted/Untrusted Language.............................................................................................. 1580Enabling and Removing PL/Perl Support...........................................................................1581Developing Functions with PL/Perl.....................................................................................1581

Greenplum MADlib Extension for Analytics.................................................................................. 1585About MADlib......................................................................................................................1585Installing MADlib................................................................................................................. 1585Upgrading MADlib...............................................................................................................1586Uninstalling MADlib.............................................................................................................1587Examples............................................................................................................................ 1587References..........................................................................................................................1593

Greenplum Partner Connector API............................................................................................... 1595Using the GPPC API..........................................................................................................1595Building a GPPC Shared Library with PGXS.....................................................................1607Registering a GPPC Function with Greenplum Database................................................. 1607Packaging and Deployment Considerations.......................................................................1608GPPC Text Function Example........................................................................................... 1609GPPC Set-Returning Function Example............................................................................ 1611

Greenplum Fuzzy String Match Extension....................................................................................1615Soundex Functions............................................................................................................. 1615Levenshtein Functions........................................................................................................1616Metaphone Functions......................................................................................................... 1616Double Metaphone Functions.............................................................................................1617Installing and Uninstalling the Fuzzy String Match Functions............................................ 1617

Summary of Greenplum Features.................................................................................................1618Greenplum SQL Standard Conformance........................................................................... 1618Greenplum and PostgreSQL Compatibility.........................................................................1620

Chapter 10: Greenplum Database UNIX Client Documentation........ 1629Greenplum Database Client Tools for UNIX.................................................................................1630

Installing the Greenplum Client Tools................................................................................ 1630Client Tools Reference.......................................................................................................1633

Greenplum Database Load Tools for UNIX..................................................................................1634Installing the Greenplum Load Tools................................................................................. 1634

Contents OpenTopic

14

Load Tools Reference........................................................................................................ 1635

Chapter 11: Greenplum Database Windows Client Documentation..1637Greenplum Database Client Tools for Windows...........................................................................1638

Installing the Greenplum Client Tools................................................................................ 1638Running the Greenplum Client Tools.................................................................................1641Client Tools Reference.......................................................................................................1642

Greenplum Database Load Tools for Windows............................................................................ 1644Installing Greenplum Loader.............................................................................................. 1644Running Greenplum Loader............................................................................................... 1646Running gpfdist as a Windows Service..............................................................................1650Loader Program Reference................................................................................................ 1651

Chapter 12: DataDirect ODBC Drivers for Pivotal Greenplum...........1652Prerequisites.................................................................................................................................. 1653Supported Client Platforms........................................................................................................... 1654Installing on Linux Systems.......................................................................................................... 1655

Configuring the Driver on Linux......................................................................................... 1656Testing the Driver Connection on Linux.............................................................................1657

Installing on Windows Systems.................................................................................................... 1658Verifying the Version on Windows..................................................................................... 1658Configuring and Testing the Driver on Windows................................................................1658

DataDirect Driver Documentation..................................................................................................1660

Chapter 13: DataDirect JDBC Driver for Pivotal Greenplum............. 1661Prerequisites.................................................................................................................................. 1662Downloading the DataDirect JDBC Driver.................................................................................... 1663Obtaining Version Details for the Driver....................................................................................... 1664Usage Information......................................................................................................................... 1665DataDirect Driver Documentation..................................................................................................1666

15

Chapter 1

Pivotal Greenplum 5.10.2 Release Notes

Updated: August, 2018

• Welcome to Pivotal Greenplum 5.10.2• Experimental Features• Differences Compared to Open Source Greenplum Database• Supported Platforms• Pivotal Greenplum Tools and Extensions Compatibility• Hadoop Distribution Compatibility• Upgrading to Greenplum Database 5.10.2• Migrating Data to Pivotal Greenplum 5.x• Pivotal Greenplum on DCA Systems• Resolved Issues• Known Issues and Limitations• Update for gp_toolkit.gp_bloat_diag Issue

Pivotal Greenplum 5.10.2 Release Notes OpenTopic

16

Welcome to Pivotal Greenplum 5.10.2Pivotal Greenplum Database is a massively parallel processing (MPP) database server that supports nextgeneration data warehousing and large-scale analytics processing. By automatically partitioning dataand running parallel queries, it allows a cluster of servers to operate as a single database supercomputerperforming tens or hundreds times faster than a traditional database. It supports SQL, MapReduce parallelprocessing, and data volumes ranging from hundreds of gigabytes, to hundreds of terabytes.

This document contains pertinent release information about Pivotal Greenplum Database 5.10.2. Forprevious versions of the release notes for Greenplum Database, go to Pivotal Greenplum DatabaseDocumentation. For information about Greenplum Database end of life, see the Pivotal Support LifecyclePolicy.

Pivotal Greenplum 5.x software is available for download from the Pivotal Greenplum page on PivotalNetwork.

Pivotal Greenplum 5.x is based on the open source Greenplum Database project code.

Important: Pivotal Support does not provide support for open source versions of GreenplumDatabase. Only Pivotal Greenplum Database is supported by Pivotal Support.

Pivotal Greenplum 5.10.2 is a patch release that resolves some issues.

https://gpdb.docs.pivotal.io/https://gpdb.docs.pivotal.io/https://pivotal.io/support/lifecycle_policyhttps://pivotal.io/support/lifecycle_policyhttps://network.pivotal.io/products/pivotal-gpdbhttps://network.pivotal.io/products/pivotal-gpdbhttp://greenplum.org/


17

Experimental FeaturesBecause Pivotal Greenplum Database is based on the open source Greenplum Database project code,it includes several experimental features to allow interested developers to experiment with their use ondevelopment systems. Feedback will help drive development of these features, and they may becomesupported in future versions of the product.

Warning: Experimental features are not recommended or supported for production deployments.These features may change in or be removed from future versions of the product based on furthertesting and feedback. Moreover, any features that may be visible in the open source code butthat are not described in the product documentation should be considered experimental andunsupported for production use.

Greenplum Database 5.10.2 includes these experimental features:

• Storage plugins for gpbackup and gprestore.

• The DD Boost storage plugin. You can specify the --plugin-config option to store a backup ona Dell EMC Data Domain storage appliance, and restore the data from the appliance. You can alsoreplicate a backup on a separate, remote Data Domain system for disaster recovery.

• The S3 storage plugin. You can specify the --plugin-config option to store a backup on anAmazon Web Services S3 location, and restore the data from the S3 location.

• Storage plugin framework API. Partners, customers, and OSS developers can develop plugins touse in conjunction with gpbackup and gprestore.

For information about storage plugins and the storage plugin API, see Using gpbackup Storage Pluginsand Backup/Restore Storage Plugin API in the Pivotal Greenplum Database Documentation.

• Recursive WITH Queries (Common Table Expressions). See WITH Queries (Common TableExpressions) in the Pivotal Greenplum Database Documentation.

• Resource groups remain an experimental feature only on the SuSE 11 platform, due to limited cgroupsfunctionality in the kernel.

SuSE 12 resolves the Linux cgroup issues that caused the performance degradation when GreenplumDatabase resource groups are enabled.

• Integration with the Pivotal Greenplum-Kafka Connector (experimental). The Connector provideshigh speed, parallel data transfer from a Kafka cluster to a Pivotal Greenplum Database cluster forbatch and streaming ETL operations. Refer to the Pivotal Greenplum-Kafka Connector (Experimental)documentation for more information about this feature.

http://greenplum.org/https://gpdb.docs.pivotal.io/https://gpdb.docs.pivotal.io/../greenplum-kafka/intro.html


18

Differences Compared to Open Source GreenplumDatabase

Pivotal Greenplum 5.x includes all of the functionality in the open source Greenplum Database project andadds:

• Product packaging and installation script.• Support for QuickLZ compression. QuickLZ compression is not provided in the open source version of

Greenplum Database due to licensing restrictions.• Support for managing Greenplum Database using Pivotal Greenplum Command Center.• Support for full text search and text analysis using Pivotal GPText.• Support for data connectors:

• Greenplum-Spark Connector• Greenplum-Informatica Connector• Greenplum-Kafka Connector• Gemfire-Greenplum Connector

• Data Direct ODBC/JDBC Drivers• gpcopy utility for copying or migrating objects between Greenplum systems.

Pivotal Greenplum 5.x does not support the following community-contributed features of open sourceGreenplum Database:

• The PXF JDBC connector.• The PXF Apache Ignite connector.

http://greenplum.org/


19

Supported PlatformsPivotal Greenplum 5.10.2 runs on the following platforms:

• Red Hat Enterprise Linux 64-bit 7.x (See the following Note)• Red Hat Enterprise Linux 64-bit 6.x• SuSE Linux Enterprise Server 64-bit 12 SP2 and SP3 with kernel version greater than 4.4.73-5. (See

the following Note)• SuSE Linux Enterprise Server 64-bit 11 SP4 (See the following Note)• CentOS 64-bit 7.x• CentOS 64-bit 6.x

Note: For the supported Linux operating systems, Pivotal Greenplum Database is supportedon system hosts using either AMD or Intel CPUs based on the x86-64 architecture. Pivotalrecommends using a homogeneous set of hardware (system hosts) in a Greenplum Databasesystem.

Important: Significant Greenplum Database performance degradation has been observed whenenabling resource group-based workload management on Red Hat 6.x, CentOS 6.x, and SuSE11 systems. This issue is caused by a Linux cgroup kernel bug. This kernel bug has been fixed inCentOS 7.x and Red Hat 7.x systems.

If you use Red Hat 6 and the performance with resource groups is acceptable for your use case,upgrade your kernel to version 2.6.32-696 or higher to benefit from other fixes to the cgroupsimplementation.

SuSE 11 does not have a kernel version that resolves this issue; resource groups are stillconsidered to be an experimental feature on this platform. Resource groups are not supported onSuSE 11 for production use. See known issue 149789783.

Pivotal Greenplum on SuSE 12 supports resource groups for production use. SuSE 12 resolves theLinux cgroup kernel issues that caused the performance degradation when Greenplum Databaseresource groups are enabled.

Note: For Greenplum Database that is installed on Red Hat Enterprise Linux 7.x or CentOS 7.xprior to 7.3, an operating system issue might cause Greenplum Database that is running largeworkloads to hang in the workload. The Greenplum Database issue is caused by Linux kernel bugs.

RHEL 7.3 and CentOS 7.3 resolves the issue.

Note: Greenplum Database on SuSE Linux Enterprise systems does not support these features.

• The PL/Perl procedural language• The gpmapreduce tool• The PL/Container language extension• The Greenplum Platform Extension Framework (PXF)

Greenplum Database support on Dell EMC DCA.

• Pivotal Greenplum Database 5.10.2 is supported on DCA systems that are running DCA softwareversion 3.4 or greater.

• Only Pivotal Greenplum Database is supported on DCA systems. Open source versions of GreenplumDatabase are not supported.

• FIPS is supported on DCA software version 3.4 and greater with Pivotal Greenplum Database 5.2.0 andgreater.

Note: These Greenplum Database releases are not certified on DCA because of an incompatibilityin configuring timezone information.


20

5.5.0, 5.6.0, 5.6.1, 5.7.0, 5.8.0

These Gree

Documents

Pivotal Greenplum Database · 2020. 7. 1. · Chapter 2: Pivotal Greenplum 5.10.1 Release Notes ... ETL Hosts for Data Loading ... 536 Loading Data with gpload.....538 Transforming