EMEA FR/2003 Richard Soundy Technical Specialist BI EMEA Business Intelligence EMEA ASIQ-M 12.5

EMEA FR/2003Richard SoundyTechnical SpecialistBI EMEA

Business Intelligence EMEA

ASIQ-M 12.5

2

EMEA FR/2003Richard SoundyTechnical SpecialistEMEA PS

Breaking News!! - 1

• ASIQ 12.5 is released!• New licensing “rules”

– Developer release (5 IP connections)– Enterprise release (Unlimited [200-1000 users])– Workgroup release (<5 CPUs)

• Platforms Supported– AIX (32 and 64 bit)– HP/UX 64 bit– Solaris 64 bit– Win 2k 32 bit

3


Breaking News!! - 2

• We will be releasing ASIQ on Linux (and Win 2003)

• Platform will initially be Itanium (64 bit Intel)

• Maybe we will release on 32 bit (but not in first release)

• Date….. Hopefully Q4 2003

4


Breaking News!! – 3

• ASIQ TPC/H benchmark results• 1. Best performance of all 4-CPU servers (any

DBMS) 1760• 2. Absolute lowest system cost of all systems (intel

and RISC) $105,382 P/P $60• 3. Absolute best price/performance of all systems• 4. Absolute lowest storage usage: competitors

used 3x-9x more storage, at 6x-84x higher cost

5


So what does this mean?

• 1. IQ is DBMS that meets strict TPC-H rules.• 2. IQ Storage efficiency: 12 disks (RAID-1 which doubled

storage size) and 2 I/O channels for IQ at a cost of $8,686. Competition used 3x-9x more storage than IQ at 6x-84x higher cost.

• 3. ”IQ saves $1M per TB of input data”: IQ storage cost is $8,686 1/6 to 1/80 of the competition cost ($55K-$760K) storage saving of $45K-$750K per 100GBè $450K-$7.6M per TB of input data.

• 4. IQ Performance: IQ-Sun has the HIGHEST performance and THE LOWEST system cost in 4-CPU, 100GB. IQ and Sun(900MHz CPU) beat competition using 1.6GHz CPUs.

• 5. Simplicity and elegance of IQ: 1 server, 1 disk array(12 disks) and 2 IO channels.

6


Goals for 12.5 - 1

Enhance Overall Functionality• Load Table

– Permit continued processing for input data errors• Introduce Full Referential Integrity• IQ Multiplex

– Remove DDL Restrictions – Simplify Architecture

• Add Miscellaneous New Features and Functions

7


Goals for 12.5 - 2

Improve VLDB Performance

• Larger Device Sizes

• IQ Multiplex improvements

• Union All View Performance

• New indexes for Date/Time data types

• Performance improvements on large (many cpu) servers

• Join Optimization for Complex Queries

8


IQ v12.5 – Topics

• Referential Integrity• New IQ Indexes• Load Table Command• IQ Multiplex 12.5

– Sybase Central Changes• Miscellaneous

– Join Optimization– New Functions– New System Procedures and Tools

9


I. Referential Integrity (RI)

• 12.4.3 - RI Phase I– Multi-column Unique HG Indexes

– Provided Full Entity Integrity–Multi-column Primary Keys

• 12.5 Introduces Referential Integrity Phase II– Multi-column Non-Unique HG indexes

10


Referential Integrity

Some Definitions:– Referential Integrity (RI)

A rule defined on a column(s) on one table that allows INSERT or UPDATE of a row only if its value for the column(s) matches a value in a column(s) of a related table

RI also includes rules that dictate actions to be taken when UPDATE or DELETE on the referenced table cause the referenced data to disappear.

11


RI Definitions

• Candidate Key is a Primary Key OR

is a column(s) with a Unique Constraint

• Foreign Key (FK) and Referenced Key A column(s) whose values must be a subset of values of

another column(s). The referenced column(s) must be a candidate key and is called the Referenced Key

12


RI Definitions

• RESTRICT Action Any UPDATE or DELETE* action on a candidate key

that causes any value upon which a foreign key depends to disappear will be denied.

An UPDATE of a foreign key which results in a value which does not match a candidate keywill be denied.

RESTRICT action is the ANSI default for RI

(* DELETE implies TRUNCATION)

13


What Was Done to Implement RI - 1

• Create and Alter Table Commands– Added column and table level RI constraints

– Using ANSI Standard Syntax

• Changed Key Length Limit to 1K (1024) bytes– Total byte limit (+1 byte overhead/column)

– Was 256 bytes in 12.4.3

• Added a Non-Unique multi-column HG index – For support of multi-column foreign keys (1K limit)

14


What Was Done to Implement RI - 2

• Modified Load Table command– For handling loads with RI enabled on tables

– Details in Load Table section of this presentation

• Added new Database Option for RI– To disable/enable RI checking in the database

15


Table Create Syntax – Primary Keys

Create Table Dept( DeptNo int Primary Key , DeptName char(10) )

Create Table Emp( EmpNo int Primary Key , EmpName char(19) , Mgr int , DeptNo int )

Unique HG index is created

16


Unique HG is created

Table Create Syntax – Primary Keyor Unique Constraint

These would beCANDIDATE KEYS


Create Table Emp( EmpNo int Unique , EmpName char(19) , Mgr int , DeptNo int )

17


Table Create Syntax – Foreign Keys


Create Table Emp( EmpNo int Primary Key , EmpName char(19) , Mgr int References Emp(EmpNo) , DeptNo int References Dept(DeptNo) )

Non-Unique HG created

FKEY on Mgr column is an example of Self-Referential Integrity

18


Table Create Syntax – Constraints

Create Table Dept( DeptNo int Primary Key iq unique(5) , DeptName char(10) not null iq unique(5) )

Create Table Emp( EmpNo int Primary Key iq unique(50) , EmpName char(19) not null iq unique(50) , Mgr int References Emp(EmpNo) , DeptNo int References Dept(DeptNo) )

19


Automatic Indexing

• Primary Key or Unique Constraint– Creates Unique HG index on key column(s)

– Can only be dropped by Alter Table

• Foreign Key Constraint– Creates Non-Unique HG index on key

columns(s)

– Can only be dropped by Alter Table

20


RI Requirements

• Candidate Key must -– be a Primary Key or have a

• Unique Constraint

• Candidate Key and Foreign Key must -– Contain Same Number of Columns– Have Identical Data Types

–Same sign, precision and scale

21


RI Enforcement – RESTRICT Action

• Foreign Keys enforce the following (else ROLLBACK):– INSERT/UPDATE on Foreign Key columns

– Value entered must exist in the Reference Table Column(s)

– UPDATE/DELETE on Referenced Column– Cannot be deleted/truncated if values exist in its

FKeys– Cannot be updated to leave orphans in its FKeys

22


RI Enforcement – Some Details

• ALTER TABLE Add Foreign Key will fail– when performed on a table with data loaded and there

is an RI violation

• LOAD TABLE– Will fail/roll back by default if there is a RI violation

– New Load Table option to reject rows in violation

• No foreign key can be associated with a candidate key that is also a foreign key

23


Only RESTRICT Action is Available

• Version 12.5 offers no other alternatives– Other RI Options not available include -

–CASCADE (delete referenced rows) and–SET NULL (Set Fkey values to NULL)

• These options may be considered for follow-on releases

24


Database Option for RI

• New Public Option to enable / disable RI ChecksDisable_RI_Check – default is OFF

• May be set to ON to bypass RI checks to permit -– Load Table

– Insert

– Update

– Delete

25


RI Load Performance Implications

• If no RI has been specified, there is no performance hit• If RI is being enforced performance will depend on …

– Type of activity (Load, Delete, Update)

– Number of Foreign Key / Candidate Relationships

– Dimension of the Foreign Key / Candidate

– Platform

• Checking occurs during Phase II of table loads• Expect 5% (max) overhead for RI checking

26


Other RI Implications

• Using RI will improve query performance !!!– Provides a wealth of information to Query

Optimizer–Improves Many to Many Joins–Multi-column indexes help in some joins (and

GROUP BY clauses)

• Now that ASIQ has RI you may need to use it– Specific situation will be addressed later

27


II. New IQ Indexes

• Multi-Column Non-Unique HG

• Date Indexes

• Word Index Enhancement

28


Multi-Column Indexes

• We have discussed non-unique multi-columns– This is a HG index

– May be created manually

• What’s new is that the optimizer can use these indexes!– Not so in 12.4.3

– This index will be used for estimates in join plans

29


Date Indexes

• New Indexes have been added to support queries on– Dates

– Time

– Datetime

• These indexes support a number of query types– Ranges searching

– Datepart searches

• You may not need to denormalize for date functionality

30


Date Index

• For use with columns defined as DATE data type– Any other data type returns an error

– CANNOT be declared Unique

• Date Index will outperform HNG for Range Queries• Will work best with LF or HG on same column

– LF & HG indexes provide distinct counts for the optimizer, and

– LF/HG are best for equality predicates

31


Date Index for Queries

• Designed to support these queries -– Datepart Functions

– Month, Day of Month, Week, Day of Week, Quarter, Year, Week in Year

– With these Operators– GT, GE, LT, LE, EQ, NE, Between

• Date Index will NOT support– Sum, Average, SumDistinct

– Min, Max

32


Date Index vs. Other Index Tests100 MM Rows – Random Dates

• Date1 = ‘1990/12/15’ (29K rows)– HG = 1 sec– DateIndex = 26 sec– HNG = 29 sec

• Date1 > ‘1993/03/10’ ( 68 MM rows) – DateIndex = 14 secs– HNG = 29 sec– HG = 200 sec

• Date1 between ‘1993/03/10’ and ‘1996/08/04’ (33 MM)

– DateIndex = 38 secs– HNG = 64 sec– HG = 211 sec

33


Time Index

• For use with columns defined as TIME data type– Any other data type returns an error

– CANNOT be declared Unique

• Time Index will outperform HNG for Range Queries• Will work best with LF or HG on same column

– LF & HG indexes provide distinct counts for the optimizer

– LF/HG are best for equality predicates

34


Time Index for Queries

• Designed to support these queries -– Date Functions

– Hour, Minute, Second– With these Operators

– GT, GE, LT, LE, EQ, NE, Between

• Will NOT support– Sum, Average, SumDistinct– Min, Max– You will need a HNG or HG for these operators

35


Datetime Index

• Similar functionality of Date and Time indexes– For Datetime data types

• You should drop existing HNG indexes and replace then with one of the new indexes on data types– Date

– Datetime

– Time

36


Datetime Indexes

The components of the index are persistent bitmaps, similar in construction to an LF Bitmap structure

Index

Day

Month

Year HourMinute

Sec.

Day of Week

Quarter of Year

Week of Year

DTTM

DATE

TIME

37


Word Index Enhancement

• The like operator will support the Word Index– Previously supported with contains operator

• Both predicates below would use the Word IndexWhere company_name contains (‘Sybase’)

andWhere company_name like ‘%Sybase%’

38


III. Load Table Command

• Load Table has been modified to -– Limit the number of Integrity Constraint violations

before a Load Table transaction will Rollback– Rows in violation are NOT inserted

– Can Log violations to a file (optional)• The existing Load Table command is otherwise

intact and will continue to support table load operationsNote:There is no change to the INSERT,

UPDATE or DELETE commands.

39


Integrity Constraints Affected by Load Table

• UNIQUE Constraint

• NULL Constraint

• DATA Value Constraint– With option CONVERSION_ERROR = ‘OFF’

• FOREIGN KEY Constraint

40


DATA CONVERSION ERROR Option

• The CONVERSION_ERROR option plays a part– This option is not new and is still supported

– Default is ON– Any data conversion causes a ROLLBACK– Does not change with the new Load Table options

• For this discussion, assume CONVERSION_ERROR option is set OFF – All data conversion violations will cause NULL to be

entered in the offending column

41


Load Table Command – 12.4.x

LOAD TABLE table-name

( LOAD-specification [, …] )

FROM ‘filename’ [, …]

QUOTES OFF

ESCAPES OFF

< other load options >

42


Load Table Command – 12.5

LOAD TABLE LOAD TABLE table-nametable-name ( LOAD-specification [, …] )( LOAD-specification [, …] ) FROM ‘filename’ [, …]FROM ‘filename’ [, …] QUOTES OFFQUOTES OFF ESCAPES OFFESCAPES OFF

< other load options >< other load options >[ IGNORE CONSTRAINT constrainttype [, …] ][ MESSAGE LOG ‘file’ ROW LOG ‘file’ [ONLY LOG logwhat,

[ …] ][ LOG DELIMITED BY ‘string’ ]

constrainttype { UNIQUE integer | NULL integer | FOREIGN KEY integer | DATA VALUE integer | ALL integer }

logwhat: { ALL | NULL | UNIQUE | DATA VALUE | FOREIGN KEY }

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

43


IGNORE CONSTRAINT Clause

[ IGNORE CONSTRAINT constrainttype [, …] ]

constrainttype { UNIQUE integer | NULL integer |

FOREIGN KEY integer | DATA VALUE integer |

ALL integer }

• When any one the constrainttypes are specified with a value– If zero, then all violations of that type are ignored– If non-zero (say N) the N+1 occurrence will cause a

ROLLBACK

44


ALL Constraints Specified

• The ALL constrainttype has a cumulative effectExample: IGNORE CONSTRAINT NULL 50, UNIQUE 100, ALL

125

• Rollback will occur when – – 50 NULL constraint violations occur, or

– 100 UNIQUE constraint violations occur, or

– Any FOREIGN KEY violation occurs, or

– 125 total violations (ALL types combined) occur

– DATA VALUE violations– Depends on CONVERSION_ERROR setting

45


Constraint Violation Logs

[ MESSAGE LOG ‘file’ ROW LOG ‘file’ [ONLY LOG logwhat, [ …] ]

[ LOG DELIMITED BY ‘string’ ]

logwhat: { ALL | NULL | UNIQUE | DATA VALUE | FOREIGN KEY }

• Options to specify what and where to log constraint violations– MESSAGE LOG (this is NOT the IQ Message Log)

– Logs Integrity Violations as specified by logwhat– ROW LOG

– Logs the ROWID and all input data values– Comma delimited (default delimiter) – Delimiter may be specified

46


MESSAGE LOG - Three Parts

• Header (Datetime load started)• Row Information (rowid, type, column number)

– ROWID– ROWID in table where row would have been loaded

– Type of Integrity Violation– One row per violation

– Column number in the table schema

• Trailer Message (Datetime load completed)

47


MESSAGE LOG - Sample

Load Table foo …Ignore Constraint UNIQUE 200Message Log ‘msg.log’ Row Log ‘row.log’Log Unique, Null, Data ValueLog Delimited By ‘|’

Message Log (msg.log) contents 2002-07-15 15:00:23 Load Table FOO: Integrity Constraint Violations

1267 DATA VALUE 43126 UNIQUE 13216 NULL 32002-07-15 16:00:10 LOAD TABLE FOO Completed

48


ROW LOG – Three Parts

• Header (Datetime load started)• Row Data (rowid, delimited data values for row)

– Binary/Varbinary data appear as ascii hexadecimal

– Date/Datetime values appear as configured by the– DATE_FORMAT or DATETIME FORMAT options

– Null values appear as NULL

– Filler fields will not appear

• Trailer Message (Datetime load completed)

49


ROW LOG - Sample

Load Table foo …Ignore Constraint UNIQUE 200Message Log ‘msg.log’ Row Log ‘row.log’Log Unique, Null, Data ValueLog Delimited By ‘|’

Row Log (row.log) Contents2002-07-15 15:00:23 Load Table FOO: Integrity Constraint Violations1267 |Mary Smith|56|M|ABCDEFG|1943/03/31|MC|3216 |John Jones|NULL|NULL|S|1945/02/28|NULL…

2002-07-15 16:00:10 LOAD TABLE FOO Completed

50


Putting both together

2002-07-15 15:00:23 Load Table FOO: Integrity Constraint Violations1267 DATA VALUE 43216 UNIQUE 13216 NULL 32002-07-15 16:00:10 LOAD TABLE FOO Completed

2002-07-15 15:00:23 Load Table FOO: Integrity Constraint Violations1267 |Mary Smith|56|M|ABCDEFG|1943/03/31|MC|3216 |John Jones|NULL|NULL|S|1945/02/28|NULL…

2002-07-15 16:00:10 LOAD TABLE FOO Completed

51


IQ Message File Entries

• Informational messages are sent to the IQ Message File when load completes– Number of Integrity Constraint violations that

occurred

– Number of input rows that were skipped

52


IV. Miscellaneous Changes in 12.5

• Device Size Changes

• New Functions

• New System Procedure – sp_iqlocks

• Join Optimization Changes

• Union All View Performance

• Other

53


Device Size Limits for 12.5

• For Raw Devices– All Platforms - 4 TB

• For File System Devices:– NT 128 GB– Solaris 512 GB, HP and IBM 64 GB

• Catalog Device Page Size Limit is 32K • May have 10,000 columns per table

– This is a recommended limit

54


New Query Functions

• Percentile Functions– Percentile_cont() – continuous distribution data– Percentile_disc() – discrete distribution data

• First and Top{n}– First – returns 1 row from a query with an Order By– Top{n} returns n rows with Order By (32767 limit)

• Ranking Functions– Rank(), dense_rank(), percent_rank(), ntile(n)– New functions over and partition_by also added

55


System Stored Procedure – sp_iqlocks

sp_iqlocks ( [connection,] [table_name,] [max_locks,] [sort_order] )

• Displays locks in the database including -– SA locks– IQ ‘Write’ locks

• Output will contain –– Connection ID, User Name, Table Name, Lock Type

–Lock Types: Shared, Exclusive, Write,Phantom, Anti-Phantom

–P, A are scan locks and have qualifiers: T, *, nnn

56


Sp_iqcontext

• Tracks a connection’s actual SQL sent to the serversp_iqcontext [conn_handle]

• Used to see what is happening on the server – remove the conn_handle and all connections are viewed

• Outputs command, number of threads and connection type (cursor/connection)

57


Join Optimization Changes

• Optimizes complex queries (many joins) faster– Ignores joins used only to decode lookup values

• New database option to limit join planning– Max_Join_Enumeration (default - TDB)

– Joins below this value will be fully enumerated– New Error Message if the limit is exceeded

58


Union ALL View Performance

• Improvements to query performance against large fact tables partitioned with a UNION ALL View– More parallelism against each arm of a view– Ability to employ more cpus– New Join Algorithms (SMPD, HPDJ) available

• Where possible specify RI between dimensions and all arms of the view– Column Cardinality in Fact Table View can be

derived from the PKEY in the Dimension Table

59


Star Schema Design

Fact Table

Dim 1

Dim 4Dim 3

Dim 2

60


Partition the Big Table using aView with UNION ALL

Big Fact Table

t1

t2

t3

Partitioned

Create View bigtable as

Select * from t1Union AllSelect * from t2Union AllSelect * from t3

Accessed by a View

61


Create RI ConstraintsDimension to Each Arm of Fact Table

Dim n

t1

t2

t3

PKEY

FKEY

View

Primary KeyCardinality Known

(Unique HG)

Foreign KeyCardinality is DerivedFrom RI with PKEY

RESULTOptimizer will know cardinality of

all FKEYs in each arm of the Fact Table!

62


Miscellaneous – Other Changes in 12.5

• Long Binary Data Type (64K)• Data Extract commands support Append/Incremental• Optimizer has better estimate of Temp Space required• DBCC per table• Many to Many table joins supported

– Using RI and multi-column indexes

• Joins to 128 tables per query

63


Changes to Users and Logins

• New Options for - – Password Expiration

– Connection Limit per user

• Locking / Unlocking a login

• New System Stored Procedure to view what SQL statement a user is running

64


Disk Striping Packed Option

• This option is now ON by defaultDISK_STRIPING_PACKED = ‘On’

• This option will make better use of available space in a database– Particularly when there has been periodic

DELETES

– It is available in 12.4.3 but is OFF by default

65


Query Plan

• Query_Plan is now on by default.

• This will allow a DBA to see the query plan of other users on the system by combining the .iqmsg file with sp_iqcontext stored procedure

• You can switch on Query_Detail, but beware of exploding the size of the .iqmsg file

66


V. IQ Multiplex – 12.5

• Functionality Changes• No longer the concept of ‘Simplex’ or ‘Multiplex’

mode• Write Server can remain up for DDL changes• Synchronization Changes• Versioning Changes• Sybase Central Changes• Dbremote processing changes

67


IQ Multiplex – Create Database

• All databases are ‘Multiplex ready’ when created– Multiplex is enabled by just adding a Query

server

• There is no longer a Top Level Share for database files– Catalog, accompanying files and executables

are local file to each server

• Sybase Central modified accordingly

68


DDL Restrictions

• Write Server may perform DDL anytime– Create/Alter/Drop Table – Create/Drop Index

– No need to change ‘mode’ or bounce the server

– Table versioning is affected **

• Query Servers still require synchronization– Synchronization method has changed

** Addressed shortly

69


Synchronization Process

• Write Server stays up– Query Server obtains a ‘Backup copy’ of catalog

while Write Server remains running

• Query Servers may be synchronized -– Individually or

– all at once

– They are bounced to restart with new catalog

70


Transaction Versioning

• All IQ Servers now run in ‘Strict’ Mode– ANSI Isolation Level 3

• New Table Versions are an object in the IQ Main Store– Query servers access new version from Main Store

– New Versions are immediately available– No outbound dbremote messages to Query Servers– DBremote message from Query Server notifies the Write

Server when all users have the new version

71


Table Versioning and DDL Actions

• When DDL is performed, Query Server table versions are “frozen”– Access to new table versions (DML) is stopped

until that server has been synchronized

• DDL activity does NOT affect users on the Write Server– They continue to have access to new table

versions

72


Sybase Central

• Overhauled to support 12.5– Many changes for Multiplex Implementation

– Synchronization

– Starting and Stopping Servers

• Many wizards have been simplified– Fewer interactions

Some Screen Shots Follow

73


No Visible Changes on Entry Screen

74


Create Database Wizard

75


All Servers Have a Multiplex Folder

This would be from a server that has not implemented Multiplex

76


Multiplex Folder Shows Server Status

Colored symbols show server status

77


Managing Servers – Stop and Start

78


Starting and Stopping Dbremote

79


Query Server Status

Yellow color status = Server Running but Out of Sync with Writer

80


Server is Not Running

Red color status = Server Not Running

81


Synchronization

82


Create Query Server Wizard

Create Query Server Wizard has been simplified

83


Create Administrative Script Option

The script files for starting and stopping processes are no longerprovided automatically.

84


DBRemote Changes

• All dbremote processes run on the Write Server– One process for each Multiplex Server

• Dbremote processes awaken every minute to check for messages from Query Servers– Query Servers advise Writer of lastest version info

– Permits Write server to release old versions

Note: Query servers cannot perform writes in Main Store to acknowledge new table versions after DML is performed

EMEA FR/2003Richard SoundyTechnical SpecialistBI EMEA

Business Intelligence EMEA

ASIQ-M 12.5 - End

Documents

EMEA FR/2003 Richard Soundy Technical Specialist BI EMEA Business Intelligence EMEA ASIQ-M 12.5