Upload
databaseguys
View
95
Download
2
Embed Size (px)
Citation preview
Copyright 2002 - BravePoint
The Progress V9 Database
John Harlow
BravePoint, Inc.
Email: [email protected]
Copyright 2002 - BravePoint
AgendaSetting the StageStorage AreasPerformanceUtilities (new and improved)SQL Engine (new)Schema (Metaschema and VST Schema)Size LimitsMiscellaneous
Copyright 2002 - BravePoint
Setting the Stage• John Harlow - Instructor, Consultant• Progress User since 1984 (V2.1)• Presenter at USA Progress Users
Conferences
Copyright 2002 - BravePoint
Setting the Stage: MeScott M. DuleckiPresenter at Explore! and regional user groupsBoard Member, Midwest MFG/PRO Users GroupPresident, Michigan Progress Users GroupPEG member 1998061901 Author of:
Safe Haven: Archiving in MFG/PRO
Copyright 2002 - BravePoint
Introduction• DBA Tools
• Progress Performance Tuning Guide (July 2002)• Progress Database Admin Guide (July 2002)• Virtual System Tables (July 2002)• V9 Database Administration (July 2002)• ProMonitor - database monitoring tool• WebProMonitor – Web Frontend for ProMonitor• Pro D&L - Accelerated Dump/Load Utility• DBA Resource Kit• STGEN - V8 to V9 Conversion Tool• All Books are Now Available Online
Copyright 2002 - BravePoint
BravePoint, Inc.Formerly United Systems, Inc (Feb 2001)Progress Services Provider
Training (Progress, QAD, Linux, and more) Performance Tuning & DB AdministrationInternet, Java, & WebSpeed InterfacesCustom Programming
Progress Experience125+ Employees3 members have used Progress since 1984Our most junior instructor has 8 years experience
Copyright 2002 - BravePoint
Who Are You?
Are you currently using the V9 Database in Production?V9.1A, V9.1B, V9.1C, V9.1D
If not when?
Copyright 2002 - BravePoint
What’s New in the V9 Database
Storage AreasPerformanceUtilities (new and improved)SQL Engine (new)Schema (Metaschema and VST Schema)Size LimitsMiscellaneous
Copyright 2002 - BravePoint
Storage Areas - What are TheySimilar concept to a Version 8 Multi-Volume
database but there can be one or more Areas in a database or a "family" of V8 Databases under one "head"
Also similar to Oracle/DB2 table spacesLogical group of database Objects (i.e. tables and
indexes)Physical group of storage units (one or more
extents on disk)
Copyright 2002 - BravePoint
Storage Areas - BenefitsRecords per Block (1 to 256) set by Area means more
efficient use of space in the Database (more details later)Multi-database transactions may be subject to logical data
corruption if 2 Phase Commit is not enabled; Storage Areas make it easier to keep all data in one database (no more split schema which is an obsolete technique)
It may be possible to improve performance by using Areas to distribute tables with heavy I/O volume over multiple disks
Copyright 2002 - BravePoint
Storage Areas - Benefits
• Areas make it possible to keep static (infrequently changed) and dynamic (heavy create and delete activity and subject to scattering) data separated
• Doing a dump/load on a particular table (or tables) now has real value
Copyright 2002 - BravePoint
Storage Area - DisadvantagesAreas make DB administration more complicated
There are NO Single Volume DatabasesEven the empty DBs are Multi-Volumeprobkup, procopy are Area ‘Sensitive’
Multiple High Water Marks (HWM) to Monitor (since there can be a variable extent for each Area) - there are 36 Areas in the MFG/PRO eB Database Structure File
Copyright 2002 - BravePoint
Storage Area - Disadvantages
A V9 DB will probably be larger than V8 because of the additional space allocated for each Area
Still can't add new Extents or Areas On-Linepromon does not have Area HWM information
Copyright 2002 - BravePoint
Storage Areas - Physical View
Area 6 Area 7 Area 8
Ext
ent
Ext
ent
Ext
ent
Ext
ent
Ext
ent
Ext
ent
Ext
ent
Ext
ent
Ext
ent
Disk Storage - Files or Raw Partitions
Ta
ble
A
Ta
ble
B
Ind
ex
A-1
Ind
ex
B-1
Ta
ble
C
Ta
ble
D
Ind
ex
C-1
Ind
ex
D-1
Ind
ex
E-1
Ta
ble
E
Area 51
Ext
ent
Ind
ex
E
Copyright 2002 - BravePoint
Storage Areas
Control Area (.db File) In V8 (and before) contained Extent path names In V9 contains _Area, _AreaExtent TablesThese tables can be queried with the 4GL
Primary Recovery Area (V8 BI Extents)2GB Limit is Gone! (it’s just much higher)
After-Image Areas (V8 AI Extents)User Defined Data Areas (V8 RM & IX Blocks)Schema Area (No V8 comparison exists)
Copyright 2002 - BravePoint
Storage AreasArea 6 is the Schema Area. It contains:
The Database Master blockObject to Area mapping informationSequence value blockSchema tables, indexes, and SQL views
It is the “default” Area May contain user data but not recommendedIf no other Area is created, Area 6 will contain all data (as
is the case after conversion from V8 with proutil conv89)V9.1D - mvsch option on proutil
Copyright 2002 - BravePoint
Storage AreasThe Physical Area layout is described in the Structure (.st)
fileAreas can have many extentsAn Extent is a separate file just like V8Only the last extent of an area may be variable sizeEach Area grows independently
Some options use the .st file to create and update DBs (prostrct, prorest, procopy)
Area contents (what tables goes to what Area) are defined in the Dictionary and/or Data Definitions (.df) - example on next page
Copyright 2002 - BravePoint
Storage Areas
ADD TABLE "Order" AREA "od1" DESCRIPTION "Order header Info" VALEXP "1 = 1" DUMP-NAME "order"
ADD INDEX "Order-Num" ON "Order" AREA "odi1" UNIQUE PRIMARY INDEX-FIELD "Order-num" ASCENDING
Copyright 2002 - BravePoint
Storage AreasA Table or Index Object can be placed in any data
AreaData Areas can contain indexes and/or tablesAn Area can be dedicated to one objectObjects cannot span AreasSimpler is better - do not go to extremesMy recommendation: 19 data Areas maximum;
Why?
Copyright 2002 - BravePoint
C o n t r o l A r e a S c h e m a A r e a R e c o v e r y A r e a
s p o r t s . d b s p o r t s . d 1 s p o r t s . b 1
A r e a 7C u s t o m e r + I n d e x e s
A r e a 8O r d e r O r d e r - l i n e s
A r e a 9O r d e r O r d e r - l i n e s I n d e x e s
O n l y
s p o r t s _ 7 . d1
s p o r t s _ 7 . d2
s p o r t s _ 8 . d1
s p o r t s _ 8 . d2
s p o r t s _ 8 . d3
s p o r t s _ 9 . d1
A r e a 1 0O t h e r T a b l e s & I n d e x e s
s p o r t s _ 1 0 . d 1 s p o r t s _ 1 0 . d 2
A f t e r I m a g i n g A r e a sE a c h E x t e n t i s 1 A r e a
s p o r t s . a 1
s p o r t s . a 2
s p o r t s a 3
Copyright 2002 - BravePoint
Records per BlockRecords per block may be different in each Area:
1, 2, 4, 8, 16, 32, 64, 128, or 256The DBKey/Recid/Rowid format varies but is transparent to
applications
Benefits Improved storage utilizationAllows use of large block size (8k) for better performance
DisadvantageHigher Rec/Blk = Lower Maximum Area Size
Copyright 2002 - BravePoint
Preparing for Storage AreasV8.3 Table and Index I/O Monitoring
-tablebase & -tablelimit -indexbase & -indexlimit_tablestat & _indexstat VSTs
Identify Static vs Dynamic vs Historical DataIdentify Large TablesIdentify tables that are frequently and easily scatteredIdentify tables with large record sizesIdentify unused indexes
Copyright 2002 - BravePoint
Options to Convert to V9proutil conv89
Usually takes less than 5 minutesAll Data & Indexes are stored in the Schema AreaWill NOT take advantage of Storage AreasMoving Tables and Indexes still leaves a Large Schema
Area (V9.1D proutil/mvsch)
Dump/LoadWill take ……longer than conv89Can take full advantage of Areas through easy
modifications to the .df file
Copyright 2002 - BravePoint
V9 Structure (.st) Files# Schema Area# Records/Block:64# Located in the /db directoryd “schema area”,64 /db
# Data Area# Area Name: data1 Area Number: 10# Records/Block:128d “data1”:10,128 /db1/data_10.d1 f 100000d “data1”:10,128 /db2/data_10.d2 f 100000d “data1”:10,128 /db3/data_10.d3
# Index Area#Area Name:index1Area Number: 20#Records/Block:Not given because intended for index blocksd “index1”:20 /db4/ f 100000d “index1”:20 /db4/
# Variable size BI extentb /bi/prod.b1
# After Image Extentsa /ai/data.a1 f 100000a /ai/data.a2 f 100000a /ai/data.a3 f 100000
Copyright 2002 - BravePoint
V9 Utilities - Table Moveproutil <db> -C tablemove <table> <area> Rebuilds the Indexes during the move since Recids
are no longer unique to a DB (but are unique to an Area)
Off-line as well as on-lineOn-line: Will Exclusive Lock the entire Table for the
duration of the moveOne Move = One Trx = Bigger BI/AI FilesMonitor the process with the _UserStatus VST; the
Table Lock can be monitored with promon or the _Lock VST
Copyright 2002 - BravePoint
V9 Utilities - Index Moveproutil <db> -C indexmove <Index def> <area>Index def: <owner>.<table>.<index>Run off-line or on-lineOn-line:
Only read operations will be allowed The table will be SHARE locked
Benefits: flexibility and availabilityMonitor the process with the _UserStatus VST
Copyright 2002 - BravePoint
V9 Utilities - Index Compactproutil <db> -C indexcompact
<owner>.<tablename>.<indexname>Combines partially filled adjacent index blocks Runs off-line or on-lineIndex Fix + Index Compact = Index RebuildBenefits:
Performance - fewer disk i/o’sReduces index sizeAvailability - index not locked
DB Corrupting Bug thru V9.1D (no patches)
Copyright 2002 - BravePoint
Enhancements to Utilities
Most utilities are now ‘Area Aware’Example: proutil idxbuild will only scan
the blocks in the Area where the table exists
proutil dbanalys/tabanalys/ixanalys can now report individual Areas (this is undocumented)proutil <db> -C dbanalys <area name>
Copyright 2002 - BravePoint
Enhancements to Utilities
procopy & probkupTarget Area Numbers must MatchArea Names don’t need to MatchTarget Records per Block must Matchprorest - the # of Areas must matchUses .st File (if one exists) in Target Directory
Copyright 2002 - BravePoint
Enhancements to Utilities
proutil truncate areaIf no Area is specified, all Areas that have no
Storage Objects have the HWM moved to ‘ground zero’
If an Area name is specified, the data is in the Area is removed (after confirmation)
In each case there is no impact on the physical extents
Copyright 2002 - BravePoint
V9 Performance ImprovementsDeferred Transaction Start
“transaction-begin” BI note write is deferredNo “transaction-begin” or “transaction-end” notes are written
for “read only” transactionsThis optimization is invisible to applications (you can’t forget
the 4gl transaction scoping rules)
Benefits:BI clusters may be reused sooner; Smaller BI log Improved performance for “empty” transactionsODBC Connections & many 4gl programs won’t cause an
open transaction
Copyright 2002 - BravePoint
V9 Read BuffersProcesses such as large reports reports can evict and
replace many shared buffersRead buffers might be able prevent this Read buffers are taken from the shared poolRead buffers remain accessible to all users When read buffers are enabled:
Read operations will use the read buffersWrite operations will use the public -B pool
Copyright 2002 - BravePoint
V9 Read BuffersNew client startup parameter -Bp n enables read
buffers per user# of buffers can be changed at runtime:
do for _myconnection transaction :find first _MyConnection._Myconn-NumSeqBuffers = 10.
end.run bad-report.p.do for _myconnection transaction :
find first _MyConnection._Myconn-NumSeqBuffers = 0.
end.
Copyright 2002 - BravePoint
V9 Read BuffersPer Client maximum is 64 (until V9.1C)
if more are requested, only 64 are granted
Total read buffers of all users cannot exceed 25% of -Bwhen this limit is reached further requests are denied and the
public buffers will be used
V9.1C -Bpmax Broker parameterProblems:
How to Allocate? VERY difficult questionNo way to monitor how many buffers in use and how they are
being used
Copyright 2002 - BravePoint
New Virtual System Tables
VSTs are Automatically Enabled in all V9 Databases
New VSTs_AreaStatus HWM Info_MyConnection Info about my connection_StatBase Reset table/index I/O
limits_UserStatus Monitor some online
utilities
Copyright 2002 - BravePoint
V9 Size LimitsDatabase size: BIG!Primary Recovery Area (BI) size: 16-32TB Maximum Number of Areas: 1,000Data Area size: Varies
16TB (1 row per block, 8k block size)8GB (256 rows per block, 1k block size)
Data Area block size: 8,192 bytesExtent size: Still 2 gigabytes until V9.1C
proutil <db> -C EnableLargeFiles
Copyright 2002 - BravePoint
V9 Size LimitsRecord size: 32,000 bytes (unchanged for now)Records per Area/Table: 2,147,483,647
Depends on Block Size & Record/Block ValueNumber of Records per block: 256Number of Indexes: 32,000Number Sequences: 2,000 (with 8k blocks)
Increased from 100Number of concurrent users: 10,000Number of concurrent transactions: 10,000
Copyright 2002 - BravePoint
What V9 is NOTMulti-Threaded BI AccessBuffer Cache (-B) per Area2GB File Size Limit Removed
Temp FilesDump (.d) filesDB, BI, and AI Extents (until V9.1C)
‘Pure’ 24 x 7 OperationSchema ChangesAdding new ExtentsReactivating indexes after proutil idxfix
Copyright 2002 - BravePoint
For More Information
BravePoint’s V9 Database Administration Jumpstart Class is designed for V6-V8 DBAs that want find out everything they need to know for moving to V9
Dan Foreman’s V9 Database Administration Jumpstart publication
Copyright 2002 - BravePoint