HPE Vertica_7.0.x Administrators Guide

Administrator's Guide

HP Vertica Analytic Database

Software Version: 7.0.x

Document Release Date: 2/20/2015

Legal Notices

WarrantyThe only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should beconstrued as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein.

The information contained herein is subject to change without notice.

Restricted Rights LegendConfidential computer software. Valid license from HP required for possession, use or copying. Consistent with FAR 12.211 and 12.212, Commercial ComputerSoftware, Computer Software Documentation, and Technical Data for Commercial Items are licensed to the U.S. Government under vendor's standard commerciallicense.

Copyright Notice Copyright 2006 - 2014 Hewlett-Packard Development Company, L.P.

Trademark NoticesAdobe is a trademark of Adobe Systems Incorporated.

Microsoft andWindows are U.S. registered trademarks of Microsoft Corporation.

UNIX is a registered trademark of TheOpenGroup.

HP Vertica Analytic Database (7.0.x) Page 2 of 997

ContentsContents 3

Administration Overview 51

Managing Licenses 52

Copying Enterprise, Evaluation, and Flex Zone License Files 52

Obtaining a License Key File 52

Understanding HP Vertica Licenses 52

License Types 53

Installing or Upgrading a License Key 54

New HP Vertica License Installations 54

HP Vertica License Renewals or Upgrades 54

Uploading or Upgrading a License Key Using Administration Tools 55

Uploading or Upgrading a License Key UsingManagement Console 55

Flex Table License Installations 56

Installing a Flex Table license using vsql 56

Installing a Flex Table license usingManagement Console 56

Viewing Your License Status 56

Examining Your License Key 57

Viewing Your License Status 57

Viewing Your License Status ThroughMC 58

Calculating the Database Size 58

How HP Vertica Estimates Raw Data Size 58

Excluding Data From Raw Data Size Estimate 58

Evaluating Data Type Footprint Size 59

Using AUDIT to Estimate Database Size 59

Monitoring Database Size for License Compliance 60

Viewing Your License Compliance Status 60

Manually Auditing Columnar Data Usage 60

Manually Auditing Flex Table Data Usage 61

Targeted Auditing 61


UsingManagement Console toMonitor License Compliance 62

Managing LicenseWarnings and Limits 62

Term LicenseWarnings and Expiration 62

Data Size LicenseWarnings and Remedies 62

If Your HP Vertica Enterprise Edition Database Size Exceeds Your Licensed Limits 63

If Your HP VerticaCommunity Edition Database Size Exceeds Your Licensed Limits 63

Configuring the Database 65

Configuration Procedure 66

IMPORTANT NOTES 66

Prepare Disk Storage Locations 67

Specifying Disk Storage Location During Installation 67

To Specify the Disk Storage LocationWhen You install: 68

Notes 68

Specifying Disk Storage Location During Database Creation 68

Notes 69

Specifying Disk Storage Location onMC 69

Configuring Disk Usage to Optimize Performance 69

Using Shared StorageWith HP Vertica 70

Viewing Database Storage Information 70

Disk Space Requirements for HP Vertica 70

Disk Space Requirements for Management Console 70

Prepare the Logical Schema Script 70

Prepare Data Files 71

How to NameData Files 71

Prepare Load Scripts 71

Create anOptional Sample Query Script 72

Create an Empty Database 73

Creating a Database Name and Password 73

Database Passwords 73

Create an Empty Database UsingMC 75

How to Create an Empty Database on anMC-managed Cluster 75

Administrator's GuideContents


Notes 76

Create a Database Using Administration Tools 77

Create the Logical Schema 78

Perform a Partial Data Load 79

Test the Database 79

Optimize Query Performance 80

Complete the Data Load 80

Test the Optimized Database 80

Set Up Incremental (Trickle) Loads 81

Implement Locales for International Data Sets 83

ICU Locale Support 83

Changing DB Locale for a Session 83

Specify the Default Locale for the Database 84

Override the Default Locale for a Session 85

Best Practices forWorking with Locales 85

Server Locale 86

vsql Client 86

ODBC Clients 86

JDBC and ADO.NET Clients 87

Notes and Restrictions 87

Change Transaction Isolation Levels 89

Notes 90

Configuration Parameters 91

Configuring HP Vertica Settings UsingMC 91

Configuring HP Vertica At the Command Line 93

General Parameters 93

TupleMover Parameters 96

EpochManagement Parameters 98

Monitoring Parameters 99

Profiling Parameters 101

Security Parameters 102



Database Designer Parameters 102

Internationalization Parameters 102

Data Collector Parameters 103

Kerberos Authentication Parameters 104

HCatalog Connector Parameters 105

Designing a Logical Schema 107

UsingMultiple Schemas 108

Multiple Schema Examples 108

UsingMultiple Private Schemas 108

Using Combinations of Private and Shared Schemas 110

Creating Schemas 110

Specifying Objects in Multiple Schemas 111

Setting Search Paths 111

Creating Objects That SpanMultiple Schemas 113

Tables in Schemas 114

About Base Tables 114

Automatic Projection Creation 114

About Temporary Tables 115

Local Temporary Tables 116

Automatic Projection Creation and Characteristics 116

Implementing Views 118

Creating Views 118

Using Views 118

Notes 120

Creating a Database Design 121

What Is a Design? 121

How Database Designer Creates a Design 122

WhoCanRunDatabase Designer 123

Granting and Enabling the DBDUSER Role 123

Allowing the DBDUSER to Run Database Designer UsingManagementConsole 124

Allowing the DBDUSER to Run Database Designer Programmatically 125



DBDUSERCapabilities and Limitations 126

Workflow for Running Database Designer 127

Specifying Parameters for Database Designer 129

Design Name 129

Design Types 129

Comprehensive Design 129

Incremental Design 130

Optimization Objectives 130

Design Tables with Sample Data 130

Design Queries 131

Query Repository 131

K-Safety for Design 131

Replicated and Unsegmented Projections 132

Replicated Projections 132

Unsegmented Projections 133

Statistics Analysis 133

Building a Design 133

Resetting a Design 134

Deploying a Design 136

Deploying Designs Using Database Designer 136

Deploying Designs Manually 137

How to Create a Design 137

UsingManagement Console to Create a Design 138

Using theWizard to Create a Design 139

Creating a DesignManually 141

Using Administration Tools to Create a Design 144

Creating Custom Designs 146

The Design Process 146

Planning Your Design 147

Design Requirements 147

Determining the Number of Projections to Use 147



Designing for K-Safety 148

Requirements for a K-Safe Physical SchemaDesign 148

Requirements for a Physical SchemaDesign with No K-Safety 149

Designing Replicated Projections for K-Safety 149

Designing Segmented Projections for K-Safety 150

Segmenting Projections 150

Creating Buddy Projections 151

Designing for Segmentation 151

Design Fundamentals 152

Writing and Deploying Custom Projections 152

Anatomy of a Projection 152

Column List and Encoding 153

BaseQuery 153

Sort Order 153

Segmentation 154

Designing Superprojections 154

MinimizingStorage Requirements 154

Maximizing Query Performance 155

Projection Design for MergeOperations 155

Maximizing Projection Performance 157

Choosing Sort Order: Best Practices 157

Combine RLE and Sort Order 157

Maximize the Advantages of RLE 158

Put Lower Cardinality Column First for Functional Dependencies 158

Sort for Merge Joins 159

Sort on Columns in Important Queries 160

Sort Columns of Equal Cardinality By Size 160

Sort Foreign Key Columns First, From Low to High Distinct Cardinality 160

Prioritizing Column Access Speed 160

Projection Examples 162

New K-Safe=2 Database 162



Creating Segmented Projections Example 162

Creating Unsegmented Projections Example 164

Adding Node to a Database 164

Creating Segmented Projections Example 165

Creating Unsegmented Projections Example 166

Implementing Security 168

Client Authentication 168

Connection Encryption 168

Client Authorization 169

Implementing Client Authentication 170

Supported Client Authentication Types 170

If YouWant Communication Layer Authentication 171

Password Authentication 172

About Password Creation andModification 172

Default Password Authentication 172

Profiles 172

How YouCreate andModify Profiles 173

Password Expiration 174

Account Locking 174

How to Unlock a Locked Account 174

Password Guidelines 175

What to Use 175

What to Avoid 176

About External Authentication 177

Setting up Your Environment to Create Authentication Records 177

About Local Password Authentication 178

How to Create Authentication Records 178

How to Create Authentication Records 178

If You DoNot Specify a Client AuthenticationMethod 179

Authentication Record Format and Rules 179

Formatting Rules 182



Configuring LDAPAuthentication 183

What You Need to Know to Configure LDAPAuthentication 183

Prerequisites for LDAPAuthentication 184

Terminology for LDAPAuthentication 184

DBADMINAuthentication Access and LDAP 185

Bind vs. Bind and Search 185

LDAPAnonymous Binding 186

Using LDAPover SSL and TLS 186

LDAPConfiguration Considerations 186

Workflow for Configuring LDAPBind 187

Workflow for Configuring LDAPBind and Search 188

ConfiguringMultipleLDAPServers 189

Configuring Ident Authentication 190

ClientAuthentication Records for Ident Authentication 190

Installing and Configuring an Ident Server 191

Example Authentication Records 192

Using an IP Range and Trust AuthenticationMethod 192

UsingMultiple Authentication Records 192

Record Order 193

How toModify Authentication Records 193

Using the Administration Tools 193

Using the ClientAuthentication Configuration Parameter 193

Examples 194

Implementing Kerberos Authentication 194

Kerberos Prerequisites 195

Configure HP Vertica for Kerberos Authentication 196

Point machines at the KDC and configure realms 200

Configure Clients for Kerberos Authentication 201

Configure ODBC and vsql Clients on Linux, HP-UX, AIX, MAC OSX, andSolaris 202

Configure ADO.NET, ODBC, and vsql Clients onWindows 204



Windows KDC on Active Directory withWindows built-in Kerberos clientand HP Vertica 205

Linux KDC withWindows-built-in Kerberos client and HP Vertica 205

ConfiguringWindows clients for Kerberos authentication 205

Authenticate and connect clients 205

Configure JDBC Clients on All Platforms 206

Determining the Client AuthenticationMethod 209

Troubleshooting Kerberos Authentication 209

Server's principal name doesn't match the host name 209

JDBC client authentication 211

Working Domain Name Service (DNS) 211

Clock synchronization 212

Encryption algorithm choices 212

Kerberos passwords 213

Using the ODBC Data Source Configuration utility 213

Implementing SSL 214

Certificate Authority 214

Public/private Keys 214

SSL Prerequisites 215

Prerequisites for SSL Server Authentication and SSL Encryption 215

Optional Prerequisites for SSL Server and Client Mutual Authentication 216

Generating SSLCertificates and Keys 216

Create a CA Private Key and Public Certificate 217

Creating the Server Private Key and Certificate 218

Create the Client Private Key and Certificate 219

Summary Illustration (Generating Certificates and Keys) 220

Set Server and Client Key and Certificate Permissions 220

JDBC Certificates 221

Summary Illustration (JDBC Certificates) 222

Generating Certificates and Keys for MC 222

Signed Certificates 223

Self-Signed Certificates 223



Importing a New Certificate toMC 224

To Import a New Certificate 224

Distributing Certificates and Keys 225

Configuring SSL 225

To Enable SSL: 225

Configuring SSL for ODBC Clients 226

SSLMode Parameter 226

SSLKeyFile Parameter 227

SSLCertFile Parameter 227

Configuring SSL for JDBC Clients 227

Setting Required Properties 227

Troubleshooting 227

Requiring SSL for Client Connections 228

Managing Users and Privileges 229

About Database Users 230

Types of Database Users 231

DBADMIN User 231

Object Owner 231

PUBLIC User 232

Creating a Database User 232

Notes 232

Example 232

Locking/unlocking a user's Database Access 233

Changing a user's Password 234

Changing a user's MC Password 234

About MC Users 235

Permission Group Types 235

MC User Types 235

Creating Users and Choosing an AuthenticationMethod 236

Default MC Users 236

Creating anMC User 236



Prerequisites 237

Create a New MC-authenticated User 237

Create a New LDAP-authenticated User 238

How MC Validates New Users 239

ManagingMC Users 239

WhoManages Users 239

What Kind of User Information You CanManage 240

About User Names 240

About Database Privileges 241

Default Privileges for All Users 241

Default Privileges for MC Users 242

Privileges Required for CommonDatabaseOperations 242

Schemas 242

Tables 242

Views 244

Projections 245

External Procedures 245

Libraries 245

User-Defined Functions 246

Sequences 246

Resource Pools 247

Users/Profiles/Roles 248

Object Visibility 248

I/OOperations 249

Comments 251

Transactions 251

Sessions 252

Tuning Operations 252

Privileges That Can BeGranted onObjects 253

Database Privileges 253

Schema Privileges 254



Schema Privileges and the Search Path 254

Table Privileges 255

Projection Privileges 256

Explicit Projection Creation and Privileges 256

Implicit Projection Creation and Privileges 257

Selecting From Projections 257

Dropping Projections 257

View Privileges 257

Sequence Privileges 258

External Procedure Privileges 259

User-Defined Function Privileges 259

Library Privileges 260

Resource Pool Privileges 260

Storage Location Privileges 260

Role, profile, and User Privileges 261

Metadata Privileges 262

I/O Privileges 263

Comment Privileges 264

Transaction Privileges 264

Session Privileges 265

Tuning Privileges 265

Granting and Revoking Privileges 265

About Superuser Privileges 265

About SchemaOwner Privileges 266

About Object Owner Privileges 266

How to Grant Privileges 267

How to Revoke Privileges 267

Privilege Ownership Chains 269

Modifying Privileges 271

Changing a Table Owner 271

Notes 271



Example 271

Table Reassignment with Sequences 273

Changing a SequenceOwner 274

Example 274

Viewing Privileges Granted onObjects 275

About Database Roles 278

Role Hierarchies 278

Creating and Using a Role 278

Roles onManagement Console 279

Types of Database Roles 280

DBADMIN Role 280

View a List of Database Superusers 281

DBDUSER Role 281

PSEUDOSUPERUSER Role 282

PUBLIC Role 282

Example 283

Default Roles for Database Users 283

Notes 284

Using Database Roles 284

Role Hierarchy 285

Example 285

Creating Database Roles 287

Deleting Database Roles 287

Granting Privileges to Roles 287

Example 287

Revoking Privileges From Roles 288

Granting Access to Database Roles 288

Example 289

Revoking Access From Database Roles 290

Example 290

Granting Administrative Access to a Role 291



Example 291

Revoking Administrative Access From aRole 292

Example 292

Enabling Roles 292

Disabling Roles 293

Viewing Enabled and Available Roles 293

Viewing NamedRoles 294

Viewing a User's Role 294

How to View aUser's Role 294

Users 295

Roles 296

Grants 296

Viewing User Roles onManagement Console 296

About MC Privileges and Roles 297

MC Permission Groups 297

MC's Configuration Privileges and Database Access 297

MC Configuration Privileges 298

MC Configuration Privileges By User Role 299

SUPER Role (mc) 300

ADMIN Role (mc) 301

About theMC Database Administrator Role 302

IT Role (mc) 303

About theMC IT (database) Role 303

NONE Role (mc) 303

MC Database Privileges 304

MC Database Privileges By Role 305

ADMIN Role (db) 306

About the ADMIN (MC configuration) Role 307

IT Role (db) 307

About the IT (MC configuration) Role 307

USER Role (db) 308



Granting Database Access toMC Users 308

Prerequisites 308

Grant a Database-Level Role to anMC user: 309

How MC Validates New Users 309

Mapping anMC User to a Database user's Privileges 310

How toMap anMC User to a Database User 310

What If YouMap theWrong Permissions 313

AddingMultiple MC Users to a Database 313

How to Find Out anMC user's Database Role 314

AddingMultiple Users toMC-managed Databases 315

Before You Start 315

How to AddMultiple Users to a Database 316

MC MappingMatrix 316


Running the Administration Tools 319

First TimeOnly 320

Between Dialogs 320

Using the Administration Tools Interface 321

Enter [Return] 321

OK - Cancel - Help 321

Menu Dialogs 322

List Dialogs 322

Form Dialogs 322

Help Buttons 323

K-Safety Support in Administration Tools 323

Notes for Remote Terminal Users 324

Using the Administration Tools Help 325

In aMenu Dialog 325

In a Dialog Box 326

Scrolling 326

Password Authentication 326



Distributing Changes Made to the Administration Tools Metadata 327

Administration Tools andManagement Console 327

Administration Tools Reference 330

Viewing Database Cluster State 330

Connecting to the Database 331

Starting the Database 332

Starting the Database UsingMC 332

Starting the Database Using the Administration Tools 332

Starting the Database At the Command Line 333

Stopping a Database 333

Error 333

Description 334

Resolution 334

Controlling Sessions 336

Notes 337

Restarting HP Vertica on Host 338

ConfigurationMenu Item 339

Creating a Database 339

Dropping a Database 341

Notes 341

Viewing a Database 342

Setting the Restart Policy 342

Best Practice for Restoring Failed Hardware 343

Installing External Procedure Executable Files 344

AdvancedMenuOptions 345

Rolling Back Database to the Last Good Epoch 345

Important note: 345

Stopping HP Vertica on Host 346

Killing the HP Vertica Process on Host 347

Upgrading an Enterprise or Evaluation License Key 348

Managing Clusters 349



Using Cluster Management 349


Administration Tools Metadata 349

Writing Administration Tools Scripts 350

Syntax 350

Parameters 350

Tools 351

Using Management Console 361

Connecting toMC 361

Managing Client Connections onMC 362

Managing Database Clusters onMC 363

Create an Empty Database UsingMC 364

How to Create an Empty Database on anMC-managed Cluster 364

Notes 365

Import an Existing Database IntoMC 366

How to Import an Existing Database on the Cluster 366

UsingMC on an AWS Cluster 367

ManagingMC Settings 367

Modifying Database-Specific Settings 367

ChangingMC or Agent Ports 368

If You Need to Change theMC Default Ports 368

How to Change the Agent Port 368

Change the Agent Port in config.py 368

Change the Agent Port onMC 369

How to Change theMC Port 369

Backing UpMC 369

TroubleshootingManagement Console 371

What You Can diagnose: 371

Viewing theMC Log 371

Exporting the User Audit Log 372

ToManually Export MC User Activity 372



RestartingMC 373

How to Restart MC Through theMC Interface (using Your browser) 373

How to Restart MC At the Command Line 373

Starting over 374

ResettingMC to Pre-Configured State 374

AvoidingMC Self-Signed Certificate Expiration 374

Operating the Database 375

Starting and Stopping the Database 375

Starting the Database 375

Starting the Database UsingMC 375

Starting the Database Using the Administration Tools 375

Starting the Database At the Command Line 376

Stopping the Database 376

Stopping a Running Database UsingMC 377

Stopping a Running Database Using the Administration Tools 377

Stopping a Running Database At the Command Line 377

Working with the HP Vertica Index Tool 379

Syntax 379

Parameters 380

Permissions 380

Controlling Expression Analysis 380

Performance and CRC 380

Running the Reindex Option 381

Running the CheckCRC Option 382

Handling CheckCRC Errors 383

Running the Checksort Option 383

Viewing Details of Index Tool Results 384

Working with Tables 387

Creating Base Tables 387

Creating Tables Using the /*+direct*/ Clause 387

Automatic Projection Creation 388



Characteristics of Default Automatic Projections 389

Creating a Table Like Another 390

Epochs and Node Recovery 391

Storage Location and Policies for New Tables 391

Simple Example 391

Using CREATE TABLE LIKE 391

Creating Temporary Tables 393

Global Temporary Tables 393

Local Temporary Tables 393

Creating a Temp Table Using the /*+direct*/ Clause 394

Characteristics of Default Automatic Projections 395

Preserving GLOBAL Temporary Table Data for a Transaction or Session 396

Specifying Column Encoding 396

Creating External Tables 397

Required Permissions for External Tables 397

COPY Statement Definition 397

Developing User-Defined Load (UDL) Functions for External Tables 398

Examples 398

Validating External Tables 398

Limiting theMaximum Number of Exceptions 399

Working with External Tables 399

Managing Resources for External Tables 399

Backing Up and Restoring External Tables 399

Using Sequences and Identity Columns in External Tables 399

Viewing External Table Definitions 400

External Table DMLSupport 400

Using External Table Values 400

Using External Tables 402

Using CREATE EXTERNAL TABLE AS COPY Statement 402

Storing HP Vertica Data in External Tables 403

Using External Tables with User-Defined Load (UDL) Functions 403



Organizing External Table Data 403

Altering Table Definitions 403

External Table Restrictions 404

Exclusive ALTER TABLE Clauses 404

Using Consecutive ALTER TABLE Commands 405

Adding Table Columns 405

Updating Associated Table Views 405

Specifying Default Expressions 406

About Using Volatile Functions 406

Updating Associated Table Views 406

Altering Table Columns 406

Adding Columns with a Default Derived Expression 407

Add a Default Column Value Derived From Another Column 407

Add a Default Column Value Derived From aUDSF 409

Changing a column's Data Type 410

Examples 410

How to Perform an Illegitimate ColumnConversion 411

Adding Constraints on Columns 413

Adding and Removing NOT NULLConstraints 413

Examples 413

Dropping a Table Column 414

Restrictions 414

Using CASCADE to Force a Drop 414

Examples 415

Moving a Table to Another Schema 416

Changing a Table Owner 416

Notes 417

Example 417

Table Reassignment with Sequences 419

Changing a SequenceOwner 419

Example 420



Renaming Tables 420

Using Rename to Swap Tables Within a Schema 421

Using Named Sequences 422

Types of Incrementing Value Objects 422

Using a Sequence with an Auto_Increment or Identity Column 423

Named Sequence Functions 423

Using DDLCommands and Functions With Named Sequences 424

Creating Sequences 424

Altering Sequences 426

Examples 426

Distributed Sequences 427

Loading Sequences 437

Creating and Instantiating a Sequence 437

Using a Sequence in an INSERT Command 437

Dropping Sequences 438

Example 438

Synchronizing Table Data with MERGE 438

Optimized Versus Non-OptimizedMERGE 439

Troubleshooting theMERGE Statement 441

Dropping and Truncating Tables 442

Dropping Tables 442

Truncating Tables 442

About Constraints 445

Adding Constraints 446

Adding ColumnConstraints with CREATE TABLE 446

Adding TwoConstraints on a Column 447

Adding a Foreign Key Constraint on a Column 447

AddingMulticolumnConstraints 448

Adding Constraints on Tables with Existing Data 449

Adding and Changing Constraints on Columns Using ALTER TABLE 449

Adding and Dropping NOT NULLColumnConstraints 450



Enforcing Constraints 450

Primary Key Constraints 451

Foreign Key Constraints 451

Examples 452

Unique Constraints 453

Not NULLConstraints 454

Dropping Constraints 456

Notes 456

Enforcing Primary Key and Foreign Key Constraints 458

Enforcing Primary Key Constraints 458

Enforcing Foreign Key Constraints 458

Detecting Constraint Violations Before You Commit Data 458

Detecting Constraint Violations 459

Fixing Constraint Violations 464

Reenabling Error Reporting 467

Working with Table Partitions 468

Differences Between Partitioning and Segmentation 468

Partition Operations 468

Defining Partitions 469

Table 3: Partitioning Expression and Results 470

Partitioning By Year andMonth 470

Restrictions on Partitioning Expressions 471

Best Practices for Partitioning 471

Dropping Partitions 471

Examples 472

Partitioning and Segmenting Data 473

Partitioning and Data Storage 475

Partitions and ROS Containers 475

Partition Pruning 475

Managing Partitions 475

Notes 477



Partitioning, Repartitioning, and Reorganizing Tables 477

Reorganizing Data After Partitioning 478

Monitoring Reorganization 478

Auto Partitioning 479

Examples 479

Eliminating Partitions 481

Making Past Partitions Eligible for Elimination 482

Verifying the ROS Merge 483

Examples 483

Moving Partitions 484

Archiving Steps 485

Preparing andMoving Partitions 485

Creating a Snapshot of the Intermediate Table 485

Copying the Config File to the Storage Location 486

Drop the Intermediate Table 486

Restoring Archived Partitions 486

Bulk Loading Data 489

Checking Data Format Before or After Loading 490

Converting Files Before Loading Data 491

Checking UTF-8 Compliance After Loading Data 491

Performing the Initial Database Load 491

Extracting Data From an Existing Database 492

Checking for Delimiter Characters in Load Data 492

Moving Data From an Existing Database to HP Vertica Nodes 493

Loading From a Local Hard Disk 493

Loading Over the Network 493

Loading FromWindows 494

Using Load Scripts 494

Using Absolute Paths in a Load Script 494

Running a Load Script 494

Using COPY and COPY LOCAL 495



Copying Data From anHP Vertica Client 496

Transforming Data During Loads 496

Understanding Transformation Requirements 496

Loading FLOAT Values 497

Using Expressions in COPY Statements 497

Handling Expression Errors 497

Transformation Example 498

Deriving Table Columns From Data File Columns 498

Specifying COPY FROMOptions 499

Loading From STDIN 500

Loading From a Specific Path 500

Loading BZIP andGZIPFiles 500

Loading withWildcards (glob) ON ANY NODE 500

Loading From a Local Client 501

Choosing a LoadMethod 501

Loading Directly intoWOS (AUTO) 501

Loading Directly to ROS (DIRECT) 502

Loading Data Incrementally (TRICKLE) 502

Loading DataWithout Committing Results (NOCOMMIT) 502

Using NOCOMMIT to Detect Constraint Violations 503

Using COPY Interactively 503

Canceling a COPY Statement 503

Specifying a COPY Parser 503

Specifying LoadMetadata 504

Interpreting Last Column End of Row Values 505

Using a Single End of Row Definition 506

Using a Delimiter and Record Terminator End of Row Definition 506

Loading UTF-8 Format Data 507

Loading Special Characters As Literals 507

Using a Custom Column Separator (DELIMITER) 508

Using a Custom ColumnOption DELIMITER 508



Defining a Null Value (NULL) 509

Loading NULL Values 509

Filling Columns with Trailing Nulls (TRAILINGNULLCOLS) 510

Attempting to Fill a NOT NULLColumnwith TRAILINGNULLCOLS 511

Changing the Default Escape Character (ESCAPE AS) 512

Eliminating Escape Character Handling 512

Delimiting Characters (ENCLOSED BY) 512

Using ENCLOSED BY for a Single Column 513

Specifying a Custom End of Record String (RECORD TERMINATOR) 514

Examples 514

Loading Native Varchar Data 515

Loading Binary (Native) Data 515

Loading Hexadecimal, Octal, and Bitstring Data 516

Hexadecimal Data 517

Octal Data 517

BitString Data 518

Examples 518

Loading Fixed-Width Format Data 519

Supported Options for Fixed-Width Data Loads 519

Using Nulls in Fixed-Width Data 519

Defining a Null Character (Statement Level) 520

Defining a Custom Record Terminator 520

Copying Fixed-Width Data 521

Skipping Content in Fixed-Width Data 521

Trimming Characters in Fixed-Width Data Loads 522

Using Padding in Fixed-Width Data Loads 523

Ignoring Columns and Fields in the Load File 524

Using the FILLER Parameter 524

FILLER Parameter Examples 524

Loading Data into Pre-Join Projections 525

Foreign and Primary Key Constraints 525



Concurrent Loads into Pre-Join Projections 526

Using Parallel Load Streams 528

Monitoring COPY Loads andMetrics 528

Using HP Vertica Functions 528

Using the CURRENT_LOAD_SOURCE() Function 529

Using the LOAD_STREAMS System Table 529

Using the STREAMNAME Parameter 529

Other LOAD_STREAMS Columns for COPY Metrics 530

Capturing Load Rejections and Exceptions 531

Using COPYParameters To Handle Rejections and Exceptions 531

Enforcing Truncating or Rejecting Rows (ENFORCELENGTH) 532

SpecifyingMaximum Rejections Before a Load Fails (REJECTMAX) 533

Aborting Data Loads for Any Error (ABORT ON ERROR) 533

Understanding Row Rejections and Rollback Errors 533

Saving Load Exceptions (EXCEPTIONS) 535

Saving Load Rejections (REJECTED DATA) 536

Saving Rejected Data to a Table 537

Rejection Records for Table Files 538

Querying a Rejection Records Table 538

Exporting the Rejected Records Table 540

COPY Rejected Data and Exception Files 541

Specifying Rejected Data and Exceptions Files 542

Saving Rejected Data and Exceptions Files to a Single Server 542

Using VSQL Variables for Rejected Data and Exceptions Files 543

COPY LOCALRejection and Exception Files 543

Specifying Rejected Data and Exceptions Files 544

Referential Integrity Load Violation 544

Trickle Loading Data 547

Using INSERT, UPDATE, and DELETE 547

WOS Overflow 547

Copying and Exporting Data 549



Moving Data Directly Between Databases 549

Creating SQL Scripts to Export Data 549

Exporting Data 550

Exporting Identity Columns 551

Examples of Exporting Data 551

Copying Data 552

Importing Identity Columns 553

Examples 553

Using Public and Private IP Networks 555

Identify the Public Network to HP Vertica 555

Identify the Database or Nodes Used for Import/Export 556

Using EXPORT Functions 557

Saving Scripts for Export Functions 557

Exporting the Catalog 558

Function Summary 558

Exporting All Catalog Objects 558

Projection Considerations 559

Exporting Database Designer Schema and Designs 559

Exporting Table Objects 559

Exporting Tables 560

Function Syntax 561

Exporting All Tables and Related Objects 561

Exporting a List Tables 561

Exporting a Single Table or Object 562

Exporting Objects 562

Function Syntax 563

Exporting All Objects 563

Exporting a List of Objects 564

Exporting a Single Object 565

Bulk Deleting and Purging Data 567

Choosing the Right Technique for Deleting Data 568



Best Practices for DELETE and UPDATE 569

Performance Considerations for DELETE and UPDATE Queries 569

Optimizing DELETEs and UPDATEs for Performance 570

Projection ColumnRequirements for Optimized Deletes 570

Optimized Deletes in Subqueries 570

Projection Sort Order for Optimizing Deletes 571

Purging Deleted Data 573

Setting a Purge Policy 573

Specifying the Time forWhich Delete Data Is Saved 574

Specifying the Number of Epochs That Are Saved 574

Disabling Purge 575

Manually Purging Data 575

Managing the Database 577

Connection Load Balancing 577

Native Connection Load Balancing Overview 577

IPVSOverview 578

ChoosingWhether to Use Native Connection Load Balancing or IPVS 578

About Native Connection Load Balancing 579

Load Balancing Schemes 580

Enabling and Disabling Native Connection Load Balancing 580

Resetting the Load Balancing State 581

Monitoring Native Connection Load Balancing 581

Determining to which Node a Client Has Connected 582

Connection Load Balancing Using IPVS 583

Configuring HP Vertica Nodes 585

Notes 585

Set Up the Loopback Interface 586

Disable Address Resolution Protocol (ARP) 587

Configuring the Directors 589

Install the HP Vertica IPVS Load Balancer Package 589

Before You Begin 589



If You Are Using Red Hat Enterprise Linux 5.x: 589

If You Are Using Red Hat Enterprise Linux 6.x: 590

Configure the HP Vertica IPVS Load Balancer 590

Public and Private IPs 591

Set up the HP Vertica IPVS Load Balancer Configuration File 592

Connecting to the Virtual IP (VIP) 593

Monitoring Shared Node Connections 594

DeterminingWhere Connections Are Going 595

Virtual IP Connection Problems 597

Issue 597

Resolution 597

Expected E-mail Messages From the Keepalived Daemon 597

Troubleshooting Keepalived Issues 598

Managing Nodes 600

Stop HP Vertica on a Node 600

Restart HP Vertica on a Node 601

Restarting HP Vertica on a node 601

Fault Groups 601

About the Fault Group Script 602

Creating Fault Groups 604

Modifying Fault Groups 605

How tomodify a fault group 606

Dropping Fault Groups 607

How to drop a fault group 607

How to remove all fault groups 607

How to add nodes back to a fault group 608

Monitoring Fault Groups 608

Monitoring fault groups through system tables 608

Monitoring fault groups throughManagement Console 608

Large Cluster 609

Control nodes on large clusters 610



Control nodes on small clusters 610

Planning a Large Cluster 610

Installing a Large Cluster 611

If you want to install a new large cluster 611

Sample rack-based cluster hosts topology 612

If you want to expand an existing cluster 614

Defining and Realigning Control Nodes on an Existing Cluster 614

Rebalancing Large Clusters 615

How to rebalance the cluster 616

How long will rebalance take? 616

Expanding the Database to a Large Cluster 616

Monitoring Large Clusters 617

Large Cluster Best Practices 617

Planning the number of control nodes 618

Allocate standby nodes 619

Plan for cluster growth 619

Write custom fault groups 619

Use segmented projections 619

Use the Database Designer 619

Elastic Cluster 620

The Elastic Cluster Scaling Factor 620

Enabling and Disabling Elastic Cluster 621

Scaling Factor Defaults 621

Viewing Scaling Factor Settings 621

Setting the Scaling Factor 622

Local Data Segmentation 622

Enabling and Disabling Local Segmentation 623

Elastic Cluster Best Practices 623

When to Enable Local Data Segmentation 624

Upgraded Database Consideration 624

Monitoring Elastic Cluster Rebalancing 624



Historical Rebalance Information 625

Adding Nodes 626

Adding Hosts to a Cluster 627

Prerequisites and Restrictions 627

Procedure to Add Hosts 627

Examples: 628

Adding Nodes to a Database 629

To Add Nodes to a Database UsingMC 629

To Add Nodes to a Database Using the Administration Tools: 629

Removing Nodes 631

Lowering the K-Safety Level to Allow for Node Removal 631

Removing Nodes From aDatabase 631

Prerequisites 632

Remove Unused Hosts From the Database UsingMC 632

Remove Unused Hosts From the Database Using the Administration Tools 632

Removing Hosts From aCluster 633

Prerequisites 633

Procedure to Remove Hosts 633

Replacing Nodes 635

Prerequisites 635

Best Practice for Restoring Failed Hardware 635

Replacing a Node Using the SameName and IP Address 636

Replacing a Failed Node Using a node with Different IP Address 637

Replacing a Functioning Node Using a Different Name and IP Address 638

Using the Administration Tools to Replace Nodes 638

Replace the Original Host with a New Host Using the Administration Tools 638

Using theManagement Console to Replace Nodes 639

Rebalancing Data Across Nodes 641

K-safety and Rebalancing 641

Rebalancing Failure and Projections 641

Permissions 642



Rebalancing Data Using the Administration Tools UI 642

Rebalancing Data UsingManagement Console 643

Rebalancing Data Using SQL Functions 643

Redistributing Configuration Files to Nodes 643

Stopping and Starting Nodes onMC 644

Managing Disk Space 645

Monitoring Disk Space Usage 645

Adding Disk Space to a Node 645

Replacing Failed Disks 647

Catalog and Data Files 647

Understanding the Catalog Directory 648

Reclaiming Disk Space From Deleted Records 650

Rebuilding a Table 650

Notes 650

Managing TupleMover Operations 651

Understanding the TupleMover 652

Moveout 652

ROS Containers 653

Mergeout 653

Mergeout of DeletionMarkers 654

Tuning the TupleMover 654

TupleMover Configuration Parameters 655

Resource Pool Settings 656

Loading Data 657

UsingMore Threads 657

Active Data Partitions 657

ManagingWorkloads 659

Statements 659

System Tables 660

The ResourceManager 660

ResourceManager Impact on Query Execution 661



Resource Pool Architecture 662

Modifying and Creating Resource Pools 662

Monitoring Resource Pools and Resource Usage By Queries 662

Examples 662

User Profiles 666

Example 666

Target Memory Determination for Queries in Concurrent Environments 668

Managing Resources At Query Run Time 668

Setting Run-Time Priority for the Resource Pool 669

Prioritizing Queries Within a Resource Pool 669

How to Set Run-Time Priority and Run-Time Priority Threshold 669

Changing Run-Time Priority of a Running Query 670

How ToChange the Run-Time Priority of a Running Query 670

Using CHANGE_RUNTIME_PRIORITY 671

Restoring ResourceManager Defaults 671

Best Practices for ManagingWorkload Resources 672

Basic Principles for Scalability and Concurrency Tuning 672

Guidelines for Setting Pool Parameters 672

Setting a Run-Time Limit for Queries 677

Example: 678

Using User-Defined Pools and User-Profiles forWorkloadManagement 679

Scenario: Periodic Batch Loads 679

Scenario: The CEOQuery 680

Scenario: Preventing Run-Away Queries 681

Scenario: Restricting Resource Usage of Ad Hoc Query Application 682

Scenario: Setting a Hard Limit on Concurrency For An Application 683

Scenario: HandlingMixedWorkloads (Batch vs. Interactive) 684

Scenario: Setting Priorities onQueries Issued By Different Users 685

Scenario: Continuous Load andQuery 686

Scenario: Prioritizing Short Queries At Run Time 687

Scenario: Dropping the Runtime Priority of LongQueries 687



Tuning the Built-In Pools 689

Scenario: Restricting HP Vertica to TakeOnly 60% of Memory 689

Scenario: Tuning for Recovery 689

Scenario: Tuning for Refresh 689

Scenario: Tuning TupleMover Pool Settings 690

Reducing Query Run Time 690

Real-Time Profiling 691

Managing System Resource Usage 692

Managing Sessions 692

Viewing Sessions 693

Interrupting and Closing Sessions 693

Controlling Sessions 694

Managing Load Streams 695

Working With Storage Locations 697

How HP Vertica Uses Storage Locations 697

Viewing Storage Locations and Policies 698

Viewing Disk Storage Information 698

Viewing Location Labels 698

Viewing Storage Tiers 699

Viewing Storage Policies 700

Adding Storage Locations 700

Planning Storage Locations 700

Adding the Location 701

Storage Location Subdirectories 702

Adding Labeled Storage Locations 702

Adding a Storage Location for USER Access 703

Altering Storage Location Use 704

USER Storage Location Restrictions 704

Effects of Altering Storage Location Use 704

Altering Location Labels 705

Adding a Location Label 705



Removing a Location Label 706

Effects of Altering a Location Label 706

Creating Storage Policies 707

Creating Policies Based on Storage Performance 707

Storage Levels and Priorities 708

Using the SET_OBJECT_STORAGE_POLICY Function 708

Effects of Creating Storage Policies 709

Moving Data Storage Locations 710

Moving Data StorageWhile Setting a Storage Policy 710

Effects of Moving a Storage Location 711

Clearing Storage Policies 711

Effects on Same-Name Storage Policies 712

Measuring Storage Performance 713

Measuring Performance on a Running HP Vertica Database 714

Measuring Performance Before a Cluster Is Set Up 714

Setting Storage Performance 714

How HP Vertica Uses Location Performance Settings 715

Using Location Performance Settings With Storage Policies 715

Dropping Storage Locations 716

Altering Storage Locations Before Dropping Them 716

Dropping USER Storage Locations 716

Retiring Storage Locations 716

Restoring Retired Storage Locations 717

Backing Up and Restoring the Database 719

Compatibility Requirements for Using vbr.py 719

Automating Regular Backups 719

Types of Backups 719

Full Backups 720

Object-Level Backups 720

Hard Link Local Backups 721

When to Back up the Database 721



Configuring Backup Hosts 721

Configuring Single-Node Database Hosts for Backup 722

Creating Configuration Files for Backup Hosts 722

Estimating Backup Host Disk Requirements 723

Estimating Log File Disk Requirements 723

Making Backup Hosts Accessible 723

Setting Up Passwordless SSH Access 724

Testing SSH Access 724

Changing the Default SSH Port on Backup Hosts 725

Increasing the SSH Maximum Connection Settings for a Backup Host 725

Copying Rsync and Python to the Backup Hosts 726

Configuring Hard Link Local Backup Hosts 726

Listing Host Names 726

Creating vbr.py Configuration Files 728

Specifying a Backup Name 728

Backing Up the Vertica Configuration File 729

SavingMultiple Restore Points 729

Specifying Full or Object-Level Backups 729

Entering the User Name 730

Saving the Account Password 730

Specifying the Backup Host and Directory 730

Saving the Configuration File 731

Continuing to Advanced Settings 731

Sample Configuration File 731

Changing the Overwrite Parameter Value 732

Configuring Required VBR Parameters 732

Sample Session Configuring Required Parameters 733

Configuring Advanced VBR Parameters 733

Example of Configuring Advanced Parameters 734

Configuring the Hard Link Local Parameter 734

Restrictions for Backup Encryption Option 735



Example Backup Configuration File 735

Using Hard File Link Local Backups 737

Planning Hard Link Local Backups 737

Specifying Backup Directory Locations 737

Understanding Hard Link Local Backups and Disaster Recovery 738

Creating Full and Incremental Backups 738

Running VbrWithout Optional Commands 739

Best Practices for Creating Backups 739

Object-Level Backups 740

Backup Locations and Storage 740

Saving Incremental Backups 740

When vbr.py Deletes Older Backups 741

Backup Directory Structure and Contents 741

Directory Tree 742

Multiple Restore Points 742

Creating Object-Level Backups 744

Invoking vbr.py Backup 744

Backup Locations and Naming 744

Best Practices for Object-Level Backups 745

Naming Conventions 745

Creating Backups Concurrently 746

Determining Backup Frequency 746

Understanding Object-Level Backup Contents 746

Making Changes After an Object-Level Backup 747

Understanding the Overwrite Parameter 747

Changing Principal and Dependent Objects 748

Considering Contraint References 748

Configuration Files for Object-Level Backups 748

Backup Epochs 749

Maximum Number of Backups 749

Creating Hard Link Local Backups 749



Specifying the Hard Link Local Backup Location 750

Creating Hard Link Local Backups for Tape Storage 750

Interrupting the Backup Utility 751

Viewing Backups 751

List Backups With vbr.py 752

Monitor database_snapshots 752

Query database_backups 752

Restoring Full Database Backups 753

Restoring theMost Recent Backup 754

Restoring an Archive 754

Attempting to Restore a Node That Is UP 755

Attempting to Restore to an Alternate Cluster 755

Restoring Object-Level Backups 755

Backup Locations 755

Cluster Requirements for Object-Level Restore 756

Restoring Objects to a Changed Cluster Topology 756

Projection Epoch After Restore 756

Catalog Locks During Backup Restore 757

Catalog Restore Events 757

Restoring Hard Link Local Backups 758

Restoring Full- andObject-Level Hard Link Local Backups 758

Avoiding OID and Epoch Conflicts 758

Transferring Backups to and From Remote Storage 759

Restoring to the SameCluster 760

Removing Backups 761

Deleting Backup Directories 761

Copying the Database to Another Cluster 762

Identifying Node Names for Target Cluster 763

Configuring the Target Cluster 763

Creating a Configuration File for CopyCluster 764

Copying the Database 765



Backup and Restore Utility Reference 766

VBR Utility Reference 766

Syntax 766

Parameters 767

VBR Configuration File Reference 767

[Misc] Miscellaneous Settings 767

[Database] Database Access Settings 769

[Transmission] Data Transmission During Backup Process 770

[Mapping] 771

Recovering the Database 773

Failure Recovery 773

Recovery Scenarios 774

Notes 775

Restarting HP Vertica on a Host 775

Restarting HP Vertica on a Host Using the Administration Tools 776

Restarting HP Vertica on a Host Using theManagement Console 776

Restarting the Database 776

Recovering the Cluster From a Backup 779

Monitoring Recovery 779

Viewing Log Files on Each Node 779

Viewing the Cluster State and Recover Status 779

Using System Tables toMonitor Recovery 780

Monitoring Cluster Status After Recovery 780

Exporting a Catalog 781

Best Practices for Disaster Recovery 781

Monitoring HP Vertica 785

Monitoring Log Files 785

When a Database Is Running 785

When the Database / Node Is Starting up 785

Rotating Log Files 786

Using Administration Tools Logrotate Utility 786



Manually Rotating Logs 786

Manually Creating Logrotate Scripts 787

Monitoring Process Status (ps) 789

Monitoring Linux Resource Usage 790

Monitoring Disk Space Usage 791

Monitoring Database Size for License Compliance 791

Viewing Your License Compliance Status 792

Manually Auditing Columnar Data Usage 792

Manually Auditing Flex Table Data Usage 793

Targeted Auditing 793

UsingManagement Console toMonitor License Compliance 793

Monitoring Shared Node Connections 794

Monitoring Elastic Cluster Rebalancing 795

Historical Rebalance Information 796

Monitoring Parameters 796

Monitoring Events 798

Event LoggingMechanisms 798

Event Severity Types 798

Event Data 802

Configuring Event Reporting 805

Configuring Reporting for Syslog 805

Enabling HP Vertica to Trap Events for Syslog 805

Defining Events to Trap for Syslog 806

Defining the SyslogFacility to Use for Reporting 807

Configuring Reporting for SNMP 808

Configuring Event Trapping for SNMP 809

ToConfigure HP Vertica to Trap Events for SNMP 809

To Enable Event Trapping for SNMP 810

ToDefineWhere HP Vertica Send Traps 810

To DefineWhich Events HP Vertica Traps 810

Verifying SNMP Configuration 811



Event Reporting Examples 812

Vertica.log 812

SNMP 812

Syslog 812

Using System Tables 814

Where System Tables Reside 814

How System Tables Are Organized 814

Querying Case-Sensitive Data in System Tables 815

Examples 816

RetainingMonitoring Information 818

Data Collector 818

Where Is DC Information retained? 818

DC Tables 819

Enabling and Disabling Data Collector 819

Viewing Current Data Retention Policy 819

Configuring Data Retention Policies 820

Working with Data Collection Logs 821

Clearing the Data Collector 822

Flushing Data Collector Logs 823

Monitoring Data Collection Components 823

Related Topics 824

Querying Data Collector Tables 824

Clearing PROJECTION_REFRESHES History 825

Monitoring Query Plan Profiles 826

Monitoring Partition Reorganization 826

Monitoring Resource Pools and Resource Usage By Queries 827

Examples 827

Monitoring Recovery 830

Viewing Log Files on Each Node 830

Viewing the Cluster State and Recover Status 831

Using System Tables toMonitor Recovery 831



Monitoring Cluster Status After Recovery 832

Monitoring HP Vertica UsingMC 833

About Chart Updates 833

ViewingMC HomePage 834

Tasks 834

Recent Databases 835

Monitoring Same-NameDatabases onMC 835

Monitoring Cluster Resources 836

Database 836

Messages 836

Performance 837

CPU/Memory Usage 837

User Query Type Distribution 837

Monitoring Cluster Nodes 838

FilteringWhat You See 838

If You don't SeeWhat You Expect 838

Monitoring Cluster CPU/Memory 839

Investigating Areas of Concern 839

Monitoring Cluster Performance 839

How to Get Metrics on Your Cluster 839

Node Colors andWhat They Mean 840

Filtering Nodes From the View 840

Monitoring System Resources 841

How up to Date Is the information? 841

Monitoring Query Activity 841

Monitoring Key Events 842

Filtering Chart Results 843

ViewingMore Detail 843

Monitoring Internal Sessions 844


Monitoring User Sessions 844



What Chart Colors Mean 844

Chart Results 845

Monitoring SystemMemory Usage 845

Types of SystemMemory 845

Monitoring System Bottlenecks 846

How MC Gathers System Bottleneck Data 846

The Components MC Reports on 846

How MC Handles Conflicts in Resources 846

Example 847

Monitoring User Query Phases 847



Monitoring Table Utilization 849


Monitoring Node Activity 850

MonitoringMC-managed DatabaseMessages 853

Message Severity 854

ViewingMessage Details 854

Search and Export Messages 854

SearchingMC-managed DatabaseMessages 854

ChangingMessage Search Criteria 855

Specifying Date Range Searches 855

FilteringMessages Client Side 856

ExportingMC-managed DatabaseMessages and Logs 856

MonitoringMC User Activity 859

Background Cleanup of Audit Records 860

Filter and Export Results 861

If You Perform a Factory Reset 861

Analyzing Workloads 862

About theWorkload Analyzer 862

Getting Tuning Recommendations Through an API 862



What andWhen 862

Record the Events 863

Observation Count and Time 864

KnowingWhat to Tune 864

The Tuning Description (recommended action) and Command 864

What a Tuning Operation Costs 864

Examples 864

Getting Recommendations From System Tables 866

UnderstandingWLA's Triggering Events 866

Getting Tuning Recommendations ThroughMC 866

UnderstandingWLA Triggering Conditions 867

Collecting Database Statistics 875

Statistics Used By theQuery Optimizer 876

How Statistics Are Collected 876

Using the ANALYZE ROW COUNTOperation 877

Using ANALYZE_STATISTICS 877

Using ANALYZE_HISTOGRAM 877

Examples 878

How Statistics Are Computed 879

How Statistics Are Reported 879

DeterminingWhen Statistics Were Last Updated 880

Reacting to Stale Statistics 884

Example 885

Canceling Statistics Collection 886

Best Practices for Statistics Collection 886

When to Gather Full Statistics 887

Save Statistics 888

Using Diagnostic Tools 889

Determining Your Version of HP Vertica 889

Collecting Diagnostics (scrutinize Command) 889

How to Run Scrutinize 894



How Scrutinize Collects and Packages Diagnostics 894

How to Upload Scrutinize Results to Support 896

Examples for the Scrutinize Command 898

Get Help with scrutinize Options 898

Collect Defaults in Your Cluster 898

Collect Information for a Database 898

Collect Information from a Specific Node 899

Use a Staging AreaOther Than /tmp 899

IncludeGzipped Log Files 899

Include aMessage in Your File 899

Send Results to Support 900

Collecting Diagnostics (diagnostics Command) 900

Syntax 900

Arguments 900

Using the Diagnostics Utility 901

Examples 901

Exporting a Catalog 902

Exporting Profiling Data 902

Syntax 902

Parameters 903

Example 903

Understanding Query Plans 904

How to Get Query Plan Information 905

How to save query plan information 906

Viewing EXPLAIN Output in Management Console 907

About the Query Plan inManagement Console 908

Expanding and Collapsing Query Paths in EXPLAINOutput 909

Clearing Query Data 909

Viewing Projection and ColumnMetadata in EXPLAIN output 909

Viewing EXPLAINOutput in vsql 910

About EXPLAIN Output 911



Textual output of query plans 911

Viewing the Statistics Query Plan Output 912

Viewing the Cost and Rows Path 914

Viewing the Projection Path 915

Viewing the Join Path 916

Outer versus inner join 917

Hash andmerge joins 917

Inequality joins 919

Event series joins 920

Viewing the Path ID 920

Viewing the Filter Path 921

Viewing the GROUP BY Paths 922

GROUPBYHASHQueryPlan Example 922

GROUPBY PIPELINED Query Plan Example 923

Partially Sorted GROUPBY Query Plan Example 924

Viewing the Sort Path 925

Viewing the Limit Path 926

Viewing the Data Redistribution Path 926

Viewing the Analytic Function Path 928

Viewing Node Down Information 929

Viewing theMERGE Path 930

Linking EXPLAIN Output to Error Messages and Profiling Information 931

Using the QUERY_PLAN_PROFILES table 933

Profiling Database Performance 935

How to Determine If Profiling Is Enabled 936

How to Enable Profiling 936

How to Disable Profiling 937

About Real-Time Profiling 938

About profiling counters 938

About query plan profiles 938

System tables withprofile data 939



What to look for in query profiles 939

Viewing Profile Data in Management Console 940

Monitoring Profiling Progress 941

Viewing Updated Profile Metrics 941

Expanding and collapsing query path profile data 942

About Profile Data in Management Console 942

Projectionmetadata 943

Query phase duration 944

Profile metrics 944

Execution events 945

Optimizer events 946

Clearing Query Data 947

Viewing Profile Data in vsql 947

How to profile a single statement 948

Real-Time Profiling Example 948

How to Use the Linux watch Command 948

How to Find Out Which Counters are Available 949

Sample views for counter information 950

Running scripts to create the sample views 950

Viewing counter values using the sample views 950

Combining sample views 950

Viewing real-time profile data 951

How to label queries for profiling 951

Label syntax 952

Profiling query plans 953

What you need for query plan profiling 953

How to get query plan status for small queries 954

How to get query plan status for large queries 955

Improving the readability of QUERY_PLAN_PROFILES output 957

Managing query profile data 958

Configuring data retention policies 958



Reacting to suboptimal query plans 958

About Locales 961

Unicode Character Encoding: UTF-8 (8-bit UCS/Unicode Transformation Format) 961

Locales 961

Notes 962

Locale Aware String Functions 962

UTF-8 String Functions 963

Locale Specification 965

Long Form 965

Syntax 965

Parameters 965

Collation Keyword Parameters 969

Notes 971

Examples 971

Short Form 972

Determining the Short Form of a Locale 972

Specifying a Short Form Locale 972

Supported Locales 973

Locale Restrictions andWorkarounds 984

Appendix: Binary File Formats 987

Creating Native Binary Format Files 987

File Signature 987

ColumnDefinitions 987

Row Data 990

Example 991

We appreciate your feedback! 997



Administration OverviewThis document describes the functions performed by an HP Vertica database administrator (DBA).Perform these tasks using only the dedicated database administrator account that was createdwhen you installed HP Vertica. The examples in this documentation set assume that theadministrative account name is dbadmin.

l To perform certain cluster configuration and administration tasks, the DBA (users of theadministrative account) must be able to supply the root password for those hosts. If thisrequirement conflicts with your organization's security policies, these functions must beperformed by your IT staff.

l If you perform administrative functions using a different account from the account providedduring installation, HP Vertica encounters file ownership problems.

l If you share the administrative account password, make sure that only one user runs theAdministration Tools at any time. Otherwise, automatic configuration propagation does notwork correctly.

l The Administration Tools require that the calling user's shell be /bin/bash. Other shells giveunexpected results and are not supported.

Administrator's GuideAdministration Overview


Managing LicensesYoumust license HP Vertica in order to use it. Hewlett-Packard supplies your license informationto you in the form of one or more license files, which encode the terms of your license. Two licensesare available:

l vlicense.dat, for columnar tables

l flextables.key, for Flex Zone flexible tables.

To prevent introducing special characters (such as line endings or file terminators) into the licensekey file, do not open the file in an editor or e-mail client. Though special characters are not alwaysvisible in an editor, their presence invalidates the license.

Copying Enterprise, Evaluation, and Flex ZoneLicense Files

For ease of HP Vertica Enterprise Edition installation, HP recommends that you copy the licensefile to /tmp/vlicense.dat on theAdministration host.

If you have a license for Flex Zone, HP recommends that you copy the license file to/opt/vertica/config/share/license.com.vertica.flextables.key.

Be careful not to change the license key file in any way when copying the file betweenWindowsand Linux, or to any other location. To help prevent applications from trying to alter the file, enclosethe license file in an archive file (such as a .zip or .tar file).

After copying the license file from one location to another, check that the copied file size is identicalto that of the one you received from HP Vertica.

Obtaining a License Key FileTo obtain an Enterprise Edition, Evaluation, or Flex Zone flex table license key, contact HP Verticaat: http://www.vertica.com/about/contact-us/

Your HP Vertica Community Edition download package includes the Community Edition license,which allows three nodes and 1TB of data. The HP Vertica Community Edition license does notexpire.

Understanding HP Vertica LicensesHP Vertica has flexible licensing terms. It can be licensed on the following bases:

l Term-based (valid until a specific date)

l Raw data size based (valid to store up to some amount of raw data)

Administrator's GuideManaging Licenses


l Both term-based and data-size-based

l Unlimited duration and data storage

l Raw data size based and a limit of 3 nodes (HP Vertica Community Edition)

Your license key has your licensing bases encoded into it. If you are unsure of your current license,you can view your license information from within HP Vertica.

License TypesHP Vertica Community Edition. You can download and start using Community Edition for free.The Community Edition license allows customers the following:

l 3 node limit

l 1 terabyte columnar table data limit

l 1 terabyte Flex table data limit

HP Vertica Enterprise Edition. You can purchase the Enterprise Edition license. The EnterpriseEdition license entitles customers to:

l No node limit

l Columnar data, amount specified by the license

l 1 terabyte Flex table data

Flex Zone. Flex Zone is a license for the flex tables technology, available in version 7.0.Customers can separately purchase and apply a Flex Zone license to their installation. The FlexZone license entitles customers to the licensed amount of Flex table data and removes the 3 noderestriction imposed by the Community Edition.

Customers whose primary goal is to work with flex tables can purchase a Flex Zone license. Whenthey purchase Flex Zone, customers receive a complimentary Enterprise License, which entitlesthem to one terabyte of columnar data and imposes no node limit.

Note: Customers who purchase a Flex Zone licensemust apply two licenses: their EnterpriseEdition license and their Flex Zone license.

Allowances CommunityEdition

EnterpriseEdition

Enterprise Edition + FlexZone

FlexZone

Node Limit 3 nodes Unlimited Unlimited Unlimited

ColumnarData

1 terabyte Per license Per license 1terabyte

Flex Data 1 terabyte 1 terabyte Per license Perlicense



Installing or Upgrading a License KeyThe steps you follow to apply your HP Vertica license key vary, depending on the type of licenseyou are applying and whether you are upgrading your license. This section describes the following:

l New HP Vertica License Installations

l HP Vertica License Renewals or Upgrades

l Flex Zone License Installations

New HP Vertica License Installations1. Copy the license key file to yourAdministration Host.

2. Ensure the license key's file permissions are set to at least 666 (read and write permissions forall users).

3. Install HP Vertica as described in the Installation Guide if you have not already done so. Theinterface prompts you for the license key file.

4. To install Community Edition, leave the default path blank and press OK. To apply yourevaluation or Enterprise Edition license, enter the absolute path of the license key file youdownloaded to your Administration Host and press OK. The first time you log in as theDatabase Administrator and run theAdministration Tools, the interface prompts you toaccept the End-User License Agreement (EULA).

Note: If you installedManagement Console, theMC administrator can point to thelocation of the license key duringManagement Console configuration.

5. ChooseView EULA to review the EULA.

6. Exit the EULA and choose Accept EULA to officially accept the EULA and continue installingthe license, or chooseReject EULA to reject the EULA and return to the AdvancedMenu.

HP Vertica License Renewals or UpgradesIf your license is expiring or you want your database to grow beyond your licensed data size, youmust renew or upgrade your license. Once you have obtained your renewal or upgraded license keyfile, you can install it using Administration Tools or Management Console.



Uploading or Upgrading a License Key UsingAdministration Tools

1. Copy the license key file to yourAdministration Host.

2. Ensure the license key's file permissions are set to at least 666 (read and write permissions forall users).

3. Start your database, if it is not already running.

4. In the Administration Tools, select Advanced > Upgrade License Key and click OK.

5. Enter the path to your new license key file and click OK. The interface prompts you to acceptthe End-User License Agreement (EULA).

6. Choose View EULA to review the EULA.

7. Exit the EULA and choose Accept EULA to officially accept the EULA and continue installingthe license, or choose Reject EULA to reject the EULA and return to the Advanced Toolsmenu.

Uploading or Upgrading a License Key UsingManagement Console

1. From your database's Overview page inManagement Console, click the License tab. TheLicense page displays. You can view your installed licenses on this page.

2. Click the Install New License button at the top of the License page.

3. Browse to the location of the license key from your local computer (where the web browser isinstalled) and upload the file.

4. Click the Apply button at the top of the page. The interface prompts you to accept the End-UserLicense Agreement (EULA).

5. Select the check box to officially accept the EULA and continue installing the license, or clickCancel to exit.

Note: As soon as you renew or upgrade your license key from either yourAdministrationHost or Management Console, HP Vertica applies the license update. No further warningsappear.



Flex Table License Installations

Installing a Flex Table license using vsql1. Install HP Vertica as described in the Installation Guide if you have not already done so.

2. Copy the Flex Zone flex tables license key file to yourAdministration Host. HP recommendsthat you copy the license file to/opt/vertica/config/share/license.com.vertica.flextables.key.

3. Start your database, if it is not already running.

4. In the Administration Tools, connect to your database.

5. At the vsql prompt, select INSTALL_LICENSE as described in the SQLReferenceManual.

=> SELECTINSTALL_LICENSE('/opt/vertica/config/share/license.com.vertica.flextables.key');

Installing a Flex Table license using ManagementConsole

1. Start Management Console.

2. From your database's Overview page inManagement Console, click the License tab. TheLicense page displays. You can view your installed licenses on this page.

3. Click the Install New License button at the top of the License page.

4. Browse to the location of the license key from your local computer (where the web browser isinstalled) and upload the file.

5. Click the Apply button at the top of the page. The interface prompts you to accept the End-UserLicense Agreement (EULA).

6. Select the check box to officially accept the EULA and continue installing the license, or clickCancel to exit.

Viewing Your License StatusHP Vertica has several functions to display your license terms and current status.



Examining Your License KeyUse the DISPLAY_LICENSE SQL function described in the SQLReferenceManual to display thelicense information. This function displays the dates for which your license is valid (or Perpetual ifyour license does not expire) and any raw data allowance. For example:

=> SELECT DISPLAY_LICENSE();DISPLAY_LICENSE

----------------------------------------------------HP Vertica Systems, Inc.

1/1/201112/31/20113050TB(1 row)

Or, use the LICENSES table described in the SQLReferenceManual to view information about allyour installed licenses. This table displays your license types, the dates for which your licenses arevalid, and the size and node limits your licenses impose. In the example below, the licenses tabledisplays the Community Edition license and the default license that controls HP Vertica's flex datacapacity.

=> SELECT * FROM licenses; \x

-[RECORD 1 ]--------+----------------------------------------license_id | 45035996273704986name | verticalicensee | Vertica Community Editionstart_date | 2011-11-22end_date | Perpetualsize | 1TBis_community_edition | tnode_restriction | 3-[RECORD 2 ]--------+----------------------------------------license_id | 45035996274085644name | com.vertica.flextablelicensee | Vertica Community Edition, Flextablestart_date | 2013-10-29end_date | Perpetualsize | 1TBis_community_edition | tnode_restriction |

You can also view the LICENSES table inManagement Console. On your database's Overviewpage inManagement Console, click the License tab. The License page displays information aboutyour installed licenses.

Viewing Your License StatusIf your license includes a raw data size allowance, HP Vertica periodically audits your database'ssize to ensure it remains compliant with the license agreement. If your license has an end date, HP



Vertica also periodically checks to see if the license has expired. You can see the result of thelatest audits using the GET_COMPLIANCE_STATUS function.

GET_COMPLIANCE_STATUS

---------------------------------------------------------------------------------Raw Data Size: 2.00GB +/- 0.003GBLicense Size : 4.000GBUtilization : 50%Audit Time : 2011-03-09 09:54:09.538704+00Compliance Status : The database is in compliance with respect to raw data size.License End Date: 04/06/2011Days Remaining: 28.59

(1 row)

Viewing Your License Status Through MCInformation about license usage is on the Settings page. SeeMonitoring Database Size for LicenseCompliance.

Calculating the Database SizeYou can use your HP Vertica software until your columnar data reaches themaximum raw datasize that the license agreement provides. This section describes when data is monitored, what datais included in the estimate, and the general methodology used to produce an estimate. For moreinformation about monitoring for data size, seeMonitoring Database Size for License Compliance.

How HP Vertica Estimates Raw Data SizeHP Vertica uses statistical sampling to calculate an accurate estimate of the raw data size of thedatabase. In this context, raw datameans the uncompressed, unfederated data stored in a singleHP Vertica database. For the purpose of license size audit and enforcement, HP Vertica evaluatesthe raw data size as if the data had been exported from the database in text format, rather than ascompressed data.

HP Vertica conducts your database size audit using statistical sampling. This method allows HPVertica to estimate the size of the database without significantly impacting database performance.The trade-off between accuracy and impact on performance is a small margin of error, inherent instatistical sampling. Reports on your database size include themargin of error, so you can assessthe accuracy of the estimate. To learnmore about simple random sampling, see Simple RandomSampling.

Excluding Data From Raw Data Size EstimateNot all data in the HP Vertica database is evaluated as part of the raw data size. Specifically, HPVertica excludes the following data:



http://www.stats.gla.ac.uk/steps/glossary/sampling.html#srs

l Multiple projections (underlying physical copies) of data from a logical database entity (table).Data appearing in multiple projections of the same table is counted only once.

l Data stored in temporary tables.

l Data accessible through external table definitions.

l Data that has been deleted, but which remains in the database. To understandmore aboutdeleting and purging data, see Purging Deleted Data.

l Data stored in theWOS.

l Data stored in system and work tables such as monitoring tables, Data Collector tables, andDatabase Designer tables.

Evaluating Data Type Footprint SizeThe data sampled for the estimate is treated as if it had been exported from the database in textformat (such as printed from vsql). This means that HP Vertica evaluates the data type footprintsizes as follows:

l Strings and binary types (CHAR, VARCHAR, BINARY, VARBINARY) are counted as theiractual size in bytes using UTF-8 encoding. NULL values are counted as 1-byte values (zerobytes for the NULL, and 1-byte for the delimiter).

l Numeric data types are counted as if they had been printed. Each digit counts as a byte, as doesany decimal point, sign, or scientific notation. For example, -123.456 counts as eight bytes (sixdigits plus the decimal point andminus sign).

l Date/time data types are counted as if they had been converted to text, including any hyphens orother separators. For example, a timestamp column containing the value for noon on July 4th,2011 would be 19 bytes. As text, vsql would print the value as 2011-07-04 12:00:00, which is 19characters, including the space between the date and the time.

Note: Each column has an additional byte for the column delimiter.

Using AUDIT to Estimate Database SizeTo supply amore accurate database size estimate than statistical sampling can provide, use theAUDIT function to perform a full audit. This function has parameters to set both the error_tolerance and confidence_level. Using one or both of these parameters increases or decreasesthe function's performance impact.

For instance, lowering the error_tolerance to zero (0) and raising the confidence_level to 100,provides themost accurate size estimate, and increases the performance impact of calling theAUDIT function. During a detailed, low error-tolerant audit, all of the data in the database is dumpedto a raw format to calculate its size. Since performing a stringent audit can significantly impactdatabase performance, never perform a full audit of a production database. See AUDIT for details.



Note: Unlike estimating raw data size using statistical sampling, a full audit performs SQLqueries on the full database contents, including the contents of theWOS.

Monitoring Database Size for License ComplianceYour HP Vertica license can include a data storage allowance. The allowance can be for columnardata, for flex table data, or for both types of data (two separate licenses). The audit()functionestimates the columnar table data size, while the audit_flex() function calculates the amount offlex table data storage. Monitoring data sizes for columnar and flex tables lets you plan either toschedule deleting old data to keep your database in compliance with your license, or to budget for alicense upgrade for additional data storage.

Note: An audit of columnar data includes any materialized columns in flex tables.

Viewing Your License Compliance StatusHP Vertica periodically runs an audit of the columnar data size to verify that your database remainscompliant with your license. You can view the results of themost recent audit by calling the GET_COMPLIANCE_STATUS function.

GET_COMPLIANCE_STATUS

---------------------------------------------------------------------------------Raw Data Size: 2.00GB +/- 0.003GBLicense Size : 4.000GBUtilization : 50%Audit Time : 2011-03-09 09:54:09.538704+00Compliance Status : The database is in compliance with respect to raw data size.License End Date: 04/06/2011Days Remaining: 28.59

(1 row)

Periodically running GET_COMPLIANCE_STATUS tomonitor your database's license status isusually enough to ensure that your database remains compliant with your license. If your databasebegins to near its columnar data allowance, you can use the other auditing functions describedbelow to determine where your database is growing and how recent deletes have affected the sizeof your database.

Manually Auditing Columnar Data UsageYou canmanually check license compliance for all columnar data in your database using theAUDIT_LICENSE_SIZE SQL function. This function performs the same audit that HP Verticaperiodically performs automatically. The AUDIT_LICENSE_SIZE SQL check runs in thebackground, so the function returns immediately. You can then view the audit results using GET_COMPLIANCE_STATUS.



Note:When you audit columnar data, the results include any flexible table virtual columns thatyou havematerialized. Materialized columns include columns that you specify when creating aflex table, and any that promote from virtual columns to real columns.

An alternative to AUDIT_LICENSE_SIZE is to use the AUDIT SQL function to audit the size of thecolumnar tables in your entire database by passing an empty string to the function. Unlike AUDIT_LICENSE_SIZE, this function operates synchronously, returning when it has estimated the size ofthe database.

=> SELECT AUDIT('');AUDIT

----------76376696

(1 row)

The size of the database is reported in bytes. The AUDIT function also allows you to control theaccuracy of the estimated database size using additional parameters. See the entry for the AUDITfunction in the SQLReferenceManual for full details. HP Vertica does not count the AUDITfunction results as an official audit. It takes no license compliance actions based on the results.

Note: The results of the AUDIT function do not include flexible table data. Use the AUDIT_FLEXfunction tomonitor data usage for your Flex Tables license .

Manually Auditing Flex Table Data UsageYou can use the AUDIT_FLEX function tomanually audit data usage for one or more flexible tables.The functionmeasures encoded, compressed data stored in ROS containers for the __raw__column of one or more flexible tables. The audit results include only virtual columns in flex tables,not data included inmaterialized columns. Temporary flex tables are not included in the audit.

Targeted AuditingIf audits determine that the columnar table estimates are unexpectedly large, consider schemas,tables, or partitions are using themost storage. You can use the AUDIT function to performtargeted audits of schemas, tables, or partitions by supplying the name of the entity whose size youwant to find. For example, to find the size of the online_sales schema in the VMart exampledatabase, run the following command:

VMart=> SELECT AUDIT('online_sales');AUDIT

----------35716504

(1 row)

You can also change the granularity of an audit to report the size of each entity in a larger entity (forexample, each table in a schema) by using the granularity argument of the AUDIT function. See theAUDIT function's entry in the SQLReferenceManual.



Using Management Console to Monitor LicenseCompliance

You can also get information about data storage of columnar data (for columnar tables and formaterialized columns in flex tables) through theManagement Console. This information is availablein the databaseOverview page, which displays a grid view of the database's overall health.

l The needle in the licensemeter adjusts to reflect the amount used inmegabytes.

l The grace period represents the term portion of the license.

l The Audit button returns the same information as the AUDIT() function in a graphicalrepresentation.

l The Details link within the License grid (next to the Audit button) provides historical informationabout license usage. This page also shows a progress meter of percent used toward yourlicense limit.

Managing License Warnings and Limits

Term License Warnings and ExpirationThe term portion of an HP Vertica license is easy tomanageyou are licensed to use HP Verticauntil a specific date. If the te

Data & Analytics

HPE Vertica_7.0.x Administrators Guide