Upload
timothy-thrower
View
219
Download
0
Tags:
Embed Size (px)
Citation preview
pr ligence Empowering Intelligence
Summary ManagementBy Advanced Queues
Paper # 419
Arup Nanda
Proligence, Inc
www.proligence.com
pr ligence Empowering Intelligence
Real Time Datawarehouses
SourceDW
Source
Source
Summary
PullLog Push
pr ligence Empowering Intelligence
Complex Query
• Query to find out the eligibility of a transaction based on data in other tables – complex joins
• Takes up too much time in the Query
pr ligence Empowering Intelligence
Actual ExampleSELECT DISTINCT NVL(C.COL1, R.COL1) AS COL1,RQ.COL2, COUNT(C.COL3) AS COUNTERFROM TAB1 R, TAB2 C, TAB3 CP, TAB4 RQ WHERE R.COL4 = C.COL4 AND C.COL1 = RQ.COL1 AND ( CP.COL5 = :b0OR (CP.COL5 = :b1 AND CP.COL6 = :b2)) AND CP.COL3 = C.COL3 AND NOT EXISTS (SELECT 1 FROM TAB5 CCL,TAB6 CL, TAB7 CHO WHERE CCL.COL7 = CL.COL7AND CL.COL8 = CHO.COL8AND CCL.COL3 = C.COL3AND CHO.COL1 = 'Y') GROUP BY NVL(C.COL1, R.COL1), RQ.COL2 ORDER BY RQ.COL2
pr ligence Empowering Intelligence
Options
• Materialized View– Fast Refresh Needed– Complex – Fast Refresh Not Allowed
• Materialized View of MVs– Too Many Levels – Difficult to Administer– Refresh Lags– Failure Prompts Calls to DBMS_MVIEW()
pr ligence Empowering Intelligence
Options
• A Summary Table to Hold the Data• Triggers Set Up on Source Tables
T1
T2
T3
T4
SUMMARYTABLE
Trigger
Trigger
Trigger
Trigger
pr ligence Empowering Intelligence
Example
• Tables– DEPT– EMP
• QuerySELECT DEPTNO, COUNT(*) EMP_COUNTSFROM EMPWHERE STATUS = 'ACTIVE'GROUP BY DEPTNO
pr ligence Empowering Intelligence
Example Summary Table
• Table DEPT_COUNTS– DEPTNO– EMP_COUNTS
DEPTNO EMP_COUNTS1 1202 3403 230
pr ligence Empowering Intelligence
LogicA Row Is Inserted
STATUS = ‘ACTIVE’?
Check the Existence of the DEPTNO in DEPT_COUNTS
Exists?
Upadate DEPT_COUNTS
Set EMP_COUNTS=EMP_COUNTS+1
Insert into DEPT_COUNTS
EMP_COUNTS=1DEPTNO=..
Y
NY
SELECT DEPTNO, COUNT(*) EMP_COUNTSFROM EMPWHERE STATUS = 'ACTIVE'GROUP BY DEPTNO
pr ligence Empowering Intelligence
Flaw
Time 0
DEPTNO EMP_COUNTS1 1202 340
pr ligence Empowering Intelligence
Flaw contd…
Time 1Session1 INSERTS into EMP (DEPTNO=3, EMPNO=5679)
DEPTNO EMP_COUNTS1 1202 340
pr ligence Empowering Intelligence
Flaw contd…
Time 2Trigger Finds No Record for DEPTNO=3INSERTS into DEPT_COUNTS
DEPTNO EMP_COUNTS1 1202 3403 1 New
pr ligence Empowering Intelligence
Flaw contd…
Time 3Session2 Inserts into EMP(DEPTNO=3, EMPNO=4567)
DEPTNO EMP_COUNTS1 1202 3403 1 Not Visible
pr ligence Empowering Intelligence
Flaw contd…
Time 4Trigger tries to INSERT into DEPT_COUNTS(DEPTNO=3, EMP_COUNTS=1)
DEPTNO EMP_COUNTS1 1202 3403 1 Primary Key Violated
WAITS!
pr ligence Empowering Intelligence
Flaw contd…
Time 5Session1 Commits!
DEPTNO EMP_COUNTS1 1202 3403 1 Row Is Present
pr ligence Empowering Intelligence
Flaw contd…
Time 6Session2 Errors Out!
DEPTNO EMP_COUNTS1 1202 3403 1 Row Is Present
Error!
pr ligence Empowering Intelligence
Problem
• Locking• Primary Key Violation• Solution
Session1
Session2
Trans1
Trans2 SummaryTable
ApplyingSession
pr ligence Empowering Intelligence
Advanced Queues
User
Producer
User
Message Payload Consumer
enqueue dequeue
pr ligence Empowering Intelligence
Complex
Producer1
Tags
Producer2
Producer3
Consumer1
Consumer2
pr ligence Empowering Intelligence
Types of AQs
• Persistent• Non Persistent• Single Producer/Consumer• Multiple Producers/Consumers• Publisher-Subscriber Model
pr ligence Empowering Intelligence
Queue Table (QT)Payload defined here
QueueException QueueDBMS_AQADMDBMS_AQ
AQ Components
pr ligence Empowering Intelligence
Solution
Table
Trigger
Queue
QueueTable
EnQueue DeQueue
SummaryTable
Payload
pr ligence Empowering Intelligence
Table
Trigger Queue
QueueTable
EnQueue
DeQueue
SummaryTable
Payload
pr ligence Empowering Intelligence
Database Prep
• Turn on the queue timer processes– aq_tm_processes = 2– ALTER SYSTEM possible too– Process Identified as QMNn
• Grant privileges– GRANT EXECUTE ON DBMS_AQADM TO SCOTT;
– GRANT EXECUTE ON DBMS_AQ TO SCOTT;
pr ligence Empowering Intelligence
Payload
• Oracle Object Type
create or replace type dept_counts_type as object
( action_type char(1), old_deptno number(2), new_deptno number(2))
Table
Trigger Queue
QueueTable
EnQueue
DeQueue
SummaryTable
Payload
pr ligence Empowering Intelligence
Processing
• Procedure PROCESS_DEPT_COUNTS()• Input Parameters
– Action – I, D, U, T– Old DeptNo– New DeptNo
Table
Trigger Queue
QueueTable
EnQueue
DeQueue
SummaryTable
Payload
pr ligence Empowering Intelligence
Queue Table
begin DBMS_AQADM.CREATE_QUEUE_TABLE ( queue_table => 'DEPT_COUNTS_QT', queue_payload_type=> 'DEPT_COUNTS_TYPE', multiple_consumers=> FALSE, storage_clause => 'TABLESPACE USR INITRANS 10 STORAGE (FREELISTS 10 FREELIST GROUPS 2)',
compatible => '8.1');end;
Table
Trigger Queue
QueueTable
EnQueue
DeQueue
SummaryTable
Payload
pr ligence Empowering Intelligence
Queue
begin DBMS_AQADM.CREATE_QUEUE ( queue_name =>'DEPT_COUNTS_Q', queue_table=>'DEPT_COUNTS_QT', max_retries=>'5', retry_delay=>'0');
dbms_aqadm.start_queue( 'DEPT_COUNTS_Q',TRUE,TRUE);
end;
Table
Trigger Queue
QueueTable
EnQueue
DeQueue
SummaryTable
Payload
pr ligence Empowering Intelligence
Enqueue Procedure
create or replace procedure enq_dept_counts_q (p_msg in dept_counts_type)as enq_opt dbms_aq.enqueue_options_t; msg_prop dbms_aq.message_properties_t; msg_id raw(16);begin sys.dbms_aq.enqueue ( 'DEPT_COUNTS_Q',enq_opt, msg_prop, p_msg, msg_id);end;
Table
Trigger Queue
QueueTable
EnQueue
DeQueue
SummaryTable
Payload
pr ligence Empowering Intelligence
Dequeue Procedure
create or replace procedure deq_dept_counts_qas deq_opt dbms_aq.dequeue_options_t; msg_prop dbms_aq.message_properties_t; payload dept_counts_type; msgid raw(16);begin loop deq_opt.wait := dbms_aq.forever; deq_opt.navigation := dbms_aq.next_message; dbms_aq.dequeue( 'DEPT_COUNTS_Q', deq_opt, msg_prop, payload, msgid); process_dept_counts(payload.action_type, payload.old_deptno, payload.new_deptno); commit; end loop;end;
Table
Trigger Queue
QueueTable
EnQueue
DeQueue
SummaryTable
Payload
pr ligence Empowering Intelligence
Trigger
create or replace trigger tr_ar_iud_emp after insert or delete or update on empfor each rowdeclare l_action char(1); l_old_deptno number(2); l_new_deptno number(2);begin enq_dept_counts_q( dept_counts_type( l_action, l_old_deptno, l_new_deptno));end;
Table
Trigger Queue
QueueTable
EnQueue
DeQueue
SummaryTable
Payload
pr ligence Empowering Intelligence
Solution Revisited
Table
Trigger
Queue
QueueTable
EnQueue DeQueue
SummaryTable
Payload
pr ligence Empowering Intelligence
Important Considerations
• Asynchronous• DecoupledTable
Trigger
Queue
QueueTable
EnQueue DeQueue
SummaryTable
Payload
Transaction 1 Transaction 2
pr ligence Empowering Intelligence
Administration
• Number of Messages in Queue
select count(*) from AQ$DEPT_COUNTS_QT where queue = 'DEPT_COUNTS_Q'
pr ligence Empowering Intelligence
Usage
• Datawarehouse• OLTP Complex Queries• Data from Heterogeneous Sources
– MQ Series• Maintaining Flattened Tables in OLTP
pr ligence Empowering Intelligence
Thank you!
www.proligence.cowww.proligence.comm
Summary ManagementBy Advanced Queues
Paper # 419