Upload
arnav
View
34
Download
0
Embed Size (px)
DESCRIPTION
Lucas Jellema (AMIS, The Netherlands). The very, very latest in oracle database development. Oracle Open World 2012, San Francisco Public Expertezed Session – Thursday 29 th November 2012. The very very very latest…. . Tom Kyte to the rescue…. - PowerPoint PPT Presentation
Citation preview
THE VERY, VERY LATEST IN ORACLE DATABASE DEVELOPMENT
Lucas Jellema (AMIS, The Netherlands)
Oracle Open World 2012, San FranciscoPublic Expertezed Session – Thursday 29th November 2012
THE VERY VERY VERY LATEST…
<Secret Code>
TOM KYTE TO THE RESCUE…
THE DATABASE IN MODERN ARCHITECTURES
Oracle Open World 2012, San Francisco
NO SQL
THE TOP-3 EARNING EMPLOYEES
• What can you say about the result of this query with respect to the question: “Who are our top three earning employees?”
A. Correct AnswerB. Sometimes correctC. Correct if there are never duplicate
salariesD. Not Correct
IN-LINE VIEWS
TOP-N QUERIES IN 12C
• Last part of a query to be evaluated – to fetch only selected rows from the result set:
– To select the next set of rows:
select *from emporderby sal descFETCH FIRST 3 ROWS ONLY;
select *from emporderby sal descOFFSET 3 FETCH NEXT 4 ROWS ONLY;
TOP-N% QUERYING
• To query for a percentage of the result set (rather than an absolute number of rows)
• And the next batch
select *from emporderby sal descFETCH FIRST 30 PERCENT ROWS ONLY;
select *from emporderby sal descOFFSET (0.3*(select count(*) from emp)) ROWS FETCH NEXT (0.3*(select count(*) from emp)) ROWS ONLY;
BOTTOM-N QUERY IN 12C
• Return only the last three rows in the ordered result set (in the proper order)
– or:
select *from emporderby sal descOFFSET ((select count(*) from emp)-3) ROWS FETCH NEXT 3 ROWS ONLY
select *from ( select * from emp order by sal asc FETCH FIRST 3 ROWS ONLY )order by sal desc;
IN-LINE PL/SQL FUNCTIONS AND PROCEDURES• Procedures are also allowed in-line• In-Line Functions and Procedures can invoke
each otherWITH procedure increment( operand in out number , incsize in number)isbegin operand:= operand + incsize;end;FUNCTION inc(value number) RETURN number IS l_value number(10):= value;BEGIN increment(l_value, 100); RETURN l_value;end;SELECT inc(sal)from emp
SPECIAL ‘BUSINESS RULE’: DEFAULT VALUE• The default values is the value that should be
inserted for a column when the client has ignored the column– not provided a value nor indicated NULL
• The default value is applied prior to the execution of the Before Row trigger– So :new.<column_value> has the value that will
be inserted– The Before Row trigger has no built in way to
telling whether the value was provided by the client or supplied as default by the database
• Default value is typically used for auditing purposes– Note: default values for columns exposed in UI
should be set in the client
COLUMN DEFAULT
• Columns can have default values– Static or literals– SQL expressions evaluating to a static– Pseudo-columns like USER and CURRENT_DATE
• DO NOT USE SYSDATE! DO NOT USE USER!– References to Application Context parameters
• sys_context(‘USERENV’, ‘IP_ADDRESS’)..– Some funny value to let the before row trigger know
that the real (complex) default must be calculated
create table citizens( name varchar2(100) default 'John Doe' , birthdate date default current_date - 1, city varchar2(50) default sys_context('KANE_CTX', 'DEFAULT_CITY' ), zipcode varchar2(8) default 'XYXYXYXYXQQ')
NEW OPTIONS WITH DEFAULT VALUEalter table empmodify (sal number(10,2) DEFAULT ON NULL 1000 )
alter table empmodify (empno number(5) NOT NULL DEFAULT ON NULL EMPNO_SEQ.NEXTVAL )
create table emp( empno NUMBER GENERATED BY DEFAULT AS IDENTITY (START WITH 100 INCREMENT BY 10), ...)
• Memory area that enables application developers to define, set, and access key/value pairs
• Rapid access in SQL and PL/SQL
• Two Application Contexts are always around:– CLIENTCONTEXT and USERENV
APPLICATION CONTEXT
Attribute Value
Attribute Value
Application Context
Attribute ValuePairs
select sys_context('USERENV', 'SESSION_USER')from dual
l_user:= sys_context('USERENV', 'SESSION_USER')
APPLICATION CONTEXT APPEARANCES• Per session (default)
– Stored in UGA, just like package state• Globally Accessible (shared across all
sessions)– Stored in SGA
• Associated with a Client Identifier– Attributes in a Globally Accessible Application
Context can explicitly be tied to the Client Identifier
– And are only accessible to sessions with that Client Identifier
TYPICAL WEB ARCHITECTURE USING CONNECTION POOL
JDBC Connection Pool
Session 1 Session 2 Session 3 Session 4
Package Aglobals
Package B Package C
PACKAGE STATE IS TIED TO DATABASE SESSION
JDBC Connection Pool
Session 1 Session 2 Session 3 Session 4
Package Aglobals
Package B Package C
globals
PACKAGE STATE IS TIED TO DATABASE SESSION – NOT WEB SESSION
JDBC Connection Pool
Session 1 Session 2 Session 3 Session 4
Package Aglobals
Package B Package C
globals
APPLICATION CONTEXT TO RETAIN STATE FOR LIGHT WEIGHT END USERS
JDBC Connection Pool
Session 1 Session 2 Session 3 Session 4
Package Aglobals
Package B Package C
globals ?
APPLICATION CONTEXT TO RETAIN STATE FOR LIGHT WEIGHT END USERS
JDBC Connection Pool
Session 1 Session 2 Session 3 Session 4
Package Aglobals
Package CGlobal Context
globals
globals
USERENV USERENV
APPLICATION CONTEXT TO RETAIN STATE FOR LIGHT WEIGHT END USERS
JDBC Connection Pool
Session 1 Session 2 Session 3 Session 4
Package Aglobals
Package CGlobal Context
globals
globals
USERENV USERENVUSERENV
PACKAGE GLOBALS: THE STATE OF THE PACKAGE IN A SESSION• This state is lost when the package is
recompiled– That is undesirable in a highly available
environmentPackage
PACKAGE GLOBALS CAN BE REPLACED BY APPLICATION CONTEXT• The Application Context is untouched by
recompilation of the package– All ‘globals’ in the application context retain
their valuesPackage
Application Context
EBR TO KILL PLANNED DOWNTIME (BECAUSE OF APPLICATION UPGRADE)
Base ReleaseRelease 2 Release 3
Application XVERSION 1
Application XVERSION 2
TIME TRAVELLING
FLASHBACK
• Introduced in 9i• Based on UNDO• Initially only for recovery• As of 11g – Total Recall option with
Flashback Data Archive– Controlled history keeping
• Look back into history– Query trends (version history)– Difference reporting– Audit trails (Replace journaling tables)
• Require trick for transaction history: WHO?• Also: when is the start of history?
OOW 2012 SESSION COMES TO THE RESCUE• CON8511 - Temporal Database Capabilities
with the Latest Generation of Database Technology
TOTAL RECALL - FLASHBACK DATA ARCHIVE IMPROVEMENTS• Complete schema evolution support: all table
definition, partitioning, and space management DDLs are supported on FDA-enabled tables.
• The metadata information for tracking transactions including the user context is now tracked. – This could mean that journaling tables are now
officially deprecated• And the current contents of journaling tables can
even be migrated to Flashback Data Archive• Introduction of SQL 2011 Valid Time Temporal
Modeling
TOTAL RECALL
• Import and export of history – Support for import and export using Data Pump
for FDA-enabled tables. Data Pump can now be used to export and import an FDA-enabled base table along with its schema-evolution metadata and historical row versions.
• Construct and manipulate the Flashback Data Archive– import user-generated history
• Restore points: Support for the use of named restore points in AS OF and versions queries has been added.
• Total Recall will (in all likelihood) be part of every edition of the database – including SE
VALID TIME TEMPORAL MODELING
• Validity (or effectivity) of facts recorded in a database is frequently specified through dates or timestamps – For example begin date and [derived] end date of a
price, membership, allocation, certificate, agreement• This valid time can differ from the transaction time
at which a record is entered into the database• Multiple entries with different, non-overlapping
valid-time periods can exist for a single entity• In 12c the notion of Valid Time is introduced into the
Oracle Database– The valid-time dimension consists of two date-time
columns specified in the table definition (create or alter)– These Valid Time columns specify the period during
which a record is valid– A table can have multiple valid_time markers
CREATING A TABLE WITH VALID TIME DIMENSION• Table with explicit valid time columns:
• Table with valid time dimension and implicit columns:
columns valid_time_start and valid_time_end (TIMESTAMP) are added implicitly
CREATE TABLE EMP( employee_number NUMBER, salary NUMBER, department_id NUMBER, name VARCHAR2(30), hiredate TIMESTAMP, firedate TIMESTAMP, PERIOD FOR user_time (hiredate, firedate));
CREATE TABLE EMP( employee_number NUMBER, salary NUMBER, department_id NUMBER, name VARCHAR2(30), PERIOD FOR contract_time);
VALID TIME AWARE FLASHBACK QUERIES• Select all employees who were employed at a
certain moment in time
• Perform all queries for records that are valid at a certain point in time
• Return all records currently (session time) valid
• Return all records (default)
SELECT * FROM EMP AS OF PERIOD FOR user_time TO_TIMESTAMP('01-JUN-2012 12.00.01 PM')
EXECUTE DBMS_FLASHBACK_ARCHIVE.enable_at_valid_time ( 'ASOF' , TO_TIMESTAMP('29-JUL-12 12.00.01 PM') );
EXECUTE DBMS_FLASHBACK_ARCHIVE.enable_at_valid_time('CURRENT');
EXECUTE DBMS_FLASHBACK_ARCHIVE.enable_at_valid_time('ALL');
DATABASE IN MODERN ARCHITECTURE
Database
Cache/Grid(L1, L2, L3)Enterprise
Service Bus
WSMobile
Services
Business Tier
Database
Standard Application
sLegacy
Applications
MULTI TIER ARCHITECTURE
Cache/Grid(L1, L2, L3)Enterprise
Service Bus
WSMobile
Services
Business Tier
Database
DB QRCNHTTP
JMX, JMX
Stored Procedures
HTTP RESTHTTP SOAP
FTP/WEBDAV
JDBCJPA (H/EL)
EncapsulationDecoupling
CachingBusiness Logic
Monitor, Trace, Audit
Authentication & Fine Grained Authorization
APPLICATION ARCHITECTURE:DRIVE APPLICATION FROM META DATA• Agility• Design Time at Run Time• Define part of the application behavior and
appearance through meta-data (outside the base source code)– The default settings are defined by developers
and deployed along with the application– Read and interpreted at run time– Manipulated and re-read
and re-interpreted at run time• Note: very similar to the way
the database operates:– Data Dictionary is the
meta-data driving the behavior of the database
Application
meta
SEPARATE BASE DATA AND CUSTOMIZED DATA• If a value is changed during site-level
implementation– Or run time customization
• It should be kept apart from the base ‘meta-data’– To prevent overwriting customized data when
the new release arrives– To allow for (temporarily) reverting to base data
• A simple solution: the Complex View with two underlying tables approach– Note: Select…
For Update Ofis not allowed
BaseValues
Customized Values
New release
ORIGINAL_NAME IO trg
REPLACE THE ORIGINAL SINGLE TABLE WITH A TWO-TABLE BASE/CUSTOM SPLIT • rename <original> to <base>• create table <customizations>
as select * from base where rownum = 0
• create or replace view <original>as select * from <customizations>union allselect * from <base> b left outer join <customizations> c on (b.id = c.id)where c.rowid is null
REPLACE THE ORIGINAL SINGLE TABLE WITH A TWO-TABLE BASE/CUSTOM SPLIT (2)
• create or replace trigger handle_insert_trginstead of insert on originalfor each row begin insert into <customizations> (col, col2,…) values(:new.col, :new.col2,…);end;
• create or replace trigger handle_update_trginstead of update on originalfor each row begin update <customizations> set col = :new.col, … where id = :new.id ; if sql%rowcount = 0 then insert into <customizations> (id, col, col2,…) (select id, :new.col, :new.col2 from base where id = :new.id); end if; end;
VERY SIMILAR TO THE ARCHITECTURE OF PLUGGABLE DATABASES
ROOT
PDB
New release of Oracle Database
APPLICATION ARCHITECTURE: NO SQL• NO SQL
– Complex SQL is hidden away inside the database
– Cache to not have to query all the time from the database
– … and to not take the overhead of a commit for not so important data
– Process first – in memory, on middle tier (BigData and CEP) - and only persist what is useful
Web Browser
RDBMS
JEE Application ServerNO SQL
SQL
QUERY RESULT CHANGE NOTIFICATION• Continuous Query Notification:
– Send an event when the result set for a query changes
– Background process calls PL/SQL Handler or Java Listener or OCI client when thecommit has occurred
– Event contains rowidof changed rows
• Used for:– Refreshing specific
data caches (middletier, global context)
– (custom) ReplicationPL/SQL
Java Listener
CONTINUOUS PROCESSING OF DATA STREAMS USING CQL• Aggregation, Spot deviation, Match on
complex patterns
WHO IS AFRAID OF RED, YELLOW AND BLUE
• Table Events– Column Seq number(5)– Column Payload varchar2(200)
SOLUTION USING LEAD
• With LEAD it is easy to compare a row with its successor(s)– As long as the pattern is fixed, LEAD will suffice
with look_ahead_events as( SELECT e.* , lead(payload) over (order by seq) next_color , lead(payload,2) over (order by seq) second_next_color FROM events e)select seqfrom look_ahead_eventswhere payload ='red' and next_color ='yellow' and second_next_color='blue'
FIND THE PATTERN RED, YELLOW AND BLUE• Using the new 12c Match Recognize operator
for finding patterns in relational dataSELECT *FROM events MATCH_RECOGNIZE ( ORDER BY seq MEASURES RED.seq AS redseq , MATCH_NUMBER() AS match_num ALL ROWS PER MATCH PATTERN (RED YELLOW BLUE) DEFINE RED AS RED.payload ='red', YELLOW AS YELLOW.payload ='yellow', BLUE AS BLUE.payload ='blue') MRORDER BY MR.redseq, MR.seq;
MATCH_RECOGNIZE FOR FINDING PATTERNS IN RELATIONAL DATA• The expression MATCH_RECOGNIZE provides
native SQL support to find patterns in sequences of rows
• Match_recognize returns Measures for selected (pattern matched) rows– Similar to MODEL clause
• Match Conditions are expressed in columns from the Table Source, aggregate functions and pattern functions FIRST, PREV, NEXT, LAST
• Patterns are regular expressions using match conditions to express a special sequence of rows satisfying the conditions
Table Source &
Where
Match_Recognize
Process and Filter
Select &Order By
DID WE EVER CONSECUTIVELY HIRE THREE EMPLOYEES IN THE SAME JOB?• Find a string of three subsequent hires where
each hire has the same job• Order by hiredate, pattern is two records that
each have the same job as their predecessor
SELECT *FROM EMPMATCH_RECOGNIZE ( ORDER BY hiredate MEASURES SAME_JOB.hiredate AS hireday , MATCH_NUMBER() AS match_num ALL ROWS PER MATCH PATTERN (SAME_JOB{3}) DEFINE SAME_JOB AS SAME_JOB.job = FIRST(SAME_JOB.job)) MR
THE SHOPPING ALGORITHM
THE SHOPPING ALGORITHM
• shopForItem Item ( String itemName) {
driveToShop;
Item item = buyItemAtShop ( itemName);
driveHomeFromShop;
return item;
}
GET THIS WEEK’S GROCERIES
getGroceries Item[] ( String[] shoppingList) {
Item[] items = new Item[ shoppingList.length];
for (int i=0; i < shoppingList.length; i++) {
items[i] = shopForItem (shoppingList[i]);
}
return items;
}
PENSION FUND – SEPTEMBER 2012
Employer
Participants
Job & Benefits
><
FETCHING THE DATA OF THE PENSION FUND FOR THE WEB APPLICATION
>< select * from employers where id = < 324>
select * from participants where employer_id = < 324>
select * from benefits where participant_id = <#>
1 record
100s records
10s records
REPORTING ON MANY EMPLOYERS
select * from employers
select * from participants where employer_id = <#>
select * from benefits where participant_id = <#>
10k records
100k records
100s records1 query
100s queries
10k queries
APPLICATION ARCHITECTURE – BULK RETRIEVE • Have the database bulk up the data retrieval• Return Ref Cursor, Types and Collections or
JSON/XML
select * from employerswhere id in <some set> select *
from participants where employer_id in <some set>
select b.* from benefits b join participants p on (p.id = b.participant_id)where p.employer_id in <some set>
Benefits Package
HTTP
JDBC
Other(Email, FTP/File,
XMPP/Chat)
SOA Suite
Oracle Service Bus
PL/SQL package
Table
AQ
View
WebLogic Server Database
Email ServerFile/FTP Server
Chat/IM XMPP Server
XMLDB
EPG
Native DB WebService
XML
Java/JEEOO
APPLICATION ARCHITECTURE – SERVICE ENABLING
Chat/IM XMPP Server
HTTP
JDBC
Other(Email, FTP/File,
XMPP/Chat)
SOA Suite
Oracle Service
Bus
DB
AQ
JMS
EJB
FileFTP
SDO
WS
http
PL/SQL package
Table
8i AQ
11g Native DB WebService
10g EPG
View
JEE Server Database
JAX-WS
ADF BC/SDO WS
EJB/JPA
Email ServerFile/FTP Server
UMS
9i XML DB
XMLTypes
XMLXML & XSD
JSON/ CSV
Ref Cursor
Types & CollJPublisher WS
utl_file, BFILE,
URITYPE
JMS Queue
XML/JSONRelational/Oracle Type
JMS
Adapters
Pojo
Java App
BUSINESS RULES
• Data Oriented Rules or Data Constraints• Declarative support in database
– For referential integrity • Order must be for a Customer
– For attribute and tuple rules• Salary must be numeric, • Hiredate may not be in the future, • End date must come after begin date
• No declarative support for complex data rules – across multiple records and tables– A department in France may not have less then
20% female employees– Order items of type weapon may not be part of
an order that ships around Christmas
BUSINESS RULES – WHERE AND HOW TO IMPLEMENT• Criteria:
– Safe– Well performant– Reusable and maintainable– Productive to implement
• Options– Client side
• JavaScript– Middle-tier
• Java, Enterprise Service Bus– Database
• Constraints and triggers are statement level – i/o transaction level
Database
RDBMS NOT ALWAYS EXCLUSIVELY ACCESSED THROUGH ONE LAYER
SOA, ESB, WebServices
Data Replication & Synchronization
Batch Bulk Processes Standard
Applications
LegacyApplications
11G VIRTUAL COLUMNS
• Add columns to a table based on an expression– Using ‘real’ columns, SQL Function and User
Defined Functions– No data is stored for Virtual
Columns, only meta-data– Virtual Columns can be
indexedVIRTUAL
alter table empADD (income AS (sal + nvl(comm,0)))
UNIQUENESS RULES USING VIRTUAL COLUMNS• Business Rule:
– Not more than one manager per department
alter table empadd constraint only_one_mgr_in_dept_ukunique (one_mgr_flag)
alter table empADD ( one_mgr_flag as ( case when job ='MANAGER' then deptno end ))
CHALLENGE: ORDERS BELONG TO A CUSTOMER IN ONE OF TWO TABLES• The Orders table contains Order records for
customers – either Dutch or Australian customers
• These customers are stored in two different tables
• Can we implement referential integrity to ensure that the order’s customer exists?
ORDER
CountryCustomer_Id….
OZ_CUSTOMER
IdName DUTCH_CUSTOMER
IdName
?
USING VIRTUAL COLUMNS IN FOREIGN KEY RELATIONS• A foreign key can be created on a Virtual
Column– That means for example we can have a single
column with some id– And two virtual columns with CASE expressions
that produce NULL or the ID value– With Foreign Keys on the Virtual Columns
ORDER
CountryCustomer_IdDutch_id (VC)Australian_id (VC)
OZ_CUSTOMER
IdName DUTCH_CUSTOMER
IdName
alter table ordersadd (australian_ctr_id as (case country when 'OZ' then customer_id end))
alter table ordersadd (dutch_ctr_id as (case country when 'NL' then customer_id end))
USING VIRTUAL COLUMNS IN FOREIGN KEY RELATIONS
ORDER
CountryCustomer_IdDutch_id (VC)Australian_id (VC)
OZ_CUSTOMER
IdName DUTCH_CUSTOMER
IdName
alter table ordersadd constraint odr_dcr_fk foreign key (dutch_ctr_id) references dutch_customer (id)
alter table ordersadd constraint odr_ocr_fk foreign key (australian_ctr_id) references oz_customer (id)
FOREIGN KEY SHOULD ONLY REFER TO CERTAIN RECORDS USING VC• Foreign Key can reference a UK based on a
Virtual Column• That allows a ‘conditional foreign key’ or a
foreign key that can only reference specific records in the referenced table– Only refer to Women in the PEOPLE table for the
Mother Foreign Key– Only refer to Values in the Domain Values table
for the Domain Name == ‘COLORS’
alter table domain_valuesadd (country_value as (case domain_name when 'COUNTRIES' then domain_value end))
alter table domain_valuesadd (country_value as (case domain_name when 'COUNTRIES' then domain_value end))
alter table domain_valuesadd (color_value as (case domain_name when 'COLORS' then domain_value end))
RESTRICTED FOREIGN KEYS USING VIRTUAL COLUMNS
CARS
IDMakeTypeColorYear
DOMAIN_VALUES
IdDomain_NameDomain_ValueColor_ValueGender_ValueOrderStatus_ValueCountry_ValueShipmentMethod_Value
alter table carsadd constraint car_clr_fk foreign key (color) references domain_values (color_value)
LACK OF WATERTIGHTNESSIN TRIGGER BASED RULE VALIDATOIN
• Statement time validation means:
• To prevent leakage we should validate at commit time– Logically correct as transaction is the logical unit– Effects from other sessions between statement and
commit are taken into account• However: Oracle unfortunately does not provide us
with a pre-commit or on-commit trigger• Workarounds:
– Dummy Table with Materialized View On Commit Refresh and Trigger on Materialized View
– Do a soft-commit by calling a package to do the actual commit – that will first do transaction level checks• Supported by a deferred check constraint that is violated by
each operation that potentially violates a business rule
DML in different session
VALIDATION
DMLvalidation
More DMLvalidation
Commit
SAFE SOLUTION: USE CUSTOM LOCKS• Prior to validating a certain business rule for
a specific record – acquire a custom lock– That identifies both Rule and Record– Using dbms_lock
• When a record is being validated for a certain rule, other sessions have to wait
• The commit (or rollback) releases all locks• Validation in a different session will include
all committed data
DML in different session
DMLvalidation
More DMLvalidation
Commit
SUMMARY
• Inline Views• Defaulting• Application Context• Flashback and the time dimension• NoSQL means smart SQL
– Cache refresh driven by change notification– Streaming analysis before persisting
• Decoupling galore– Bulk retrieval– Service enabling
• Business Rules• EBR• 12c promises even more