Upload
others
View
5
Download
0
Embed Size (px)
Citation preview
Still using Windows 3.1? So why stick with
SQL-92?
@ModernSQL - https://modern-sql.com/ @MarkusWinand
SQL:1999
WITH (Common Table Expressions)
Understand
this first
WITH (non-recursive) The ProblemNested queries are hard to read:
SELECT…FROM(SELECT…FROMt1JOIN(SELECT…FROM…)aON(…))bJOIN(SELECT…FROM…)cON(…)
Then this...
WITH (non-recursive) The ProblemNested queries are hard to read:
SELECT…FROM(SELECT…FROMt1JOIN(SELECT…FROM…)aON(…))bJOIN(SELECT…FROM…)cON(…)
Then this...
WITH (non-recursive) The ProblemNested queries are hard to read:
SELECT…FROM(SELECT…FROMt1JOIN(SELECT…FROM…)aON(…))bJOIN(SELECT…FROM…)cON(…)
Finally the first line makes sense
WITH (non-recursive) The ProblemNested queries are hard to read:
SELECT…FROM(SELECT…FROMt1JOIN(SELECT…FROM…)aON(…))bJOIN(SELECT…FROM…)cON(…)
CTEs are statement-scoped views:
WITHa(c1,c2,c3)AS(SELECTc1,c2,c3FROM…),
b(c4,…)AS(SELECTc4,…FROMt1JOINaON(…)),
c(…)
Keyword
WITH (non-recursive) Since SQL:1999
CTEs are statement-scoped views:
WITHa(c1,c2,c3)AS(SELECTc1,c2,c3FROM…),
b(c4,…)AS(SELECTc4,…FROMt1JOINaON(…)),
c(…)
Name of CTE and (here optional) column names
WITH (non-recursive) Since SQL:1999
CTEs are statement-scoped views:
WITHa(c1,c2,c3)AS(SELECTc1,c2,c3FROM…),
b(c4,…)AS(SELECTc4,…FROMt1JOINaON(…)),
c(…)
Definition
WITH (non-recursive) Since SQL:1999
CTEs are statement-scoped views:
WITHa(c1,c2,c3)AS(SELECTc1,c2,c3FROM…),
b(c4,…)AS(SELECTc4,…FROMt1JOINaON(…)),
c(…)
Introduces another CTE
Don't repeat WITH
WITH (non-recursive) Since SQL:1999
CTEs are statement-scoped views:
WITHa(c1,c2,c3)AS(SELECTc1,c2,c3FROM…),
b(c4,…)AS(SELECTc4,…FROMt1JOINaON(…)),
c(…)
WITH (non-recursive) Since SQL:1999
CTEs are statement-scoped views:
WITHa(c1,c2,c3)AS(SELECTc1,c2,c3FROM…),
b(c4,…)AS(SELECTc4,…FROMt1JOINaON(…)),
c(…)
May refer toprevious CTEs
WITH (non-recursive) Since SQL:1999
WITHa(c1,c2,c3)AS(SELECTc1,c2,c3FROM…),
b(c4,…)AS(SELECTc4,…FROMt1JOINaON(…)),
c(…)AS(SELECT…FROM…)
SELECT…FROMbJOINcON(…)
Third CTE
WITH (non-recursive) Since SQL:1999
WITHa(c1,c2,c3)AS(SELECTc1,c2,c3FROM…),
b(c4,…)AS(SELECTc4,…FROMt1JOINaON(…)),
c(…)AS(SELECT…FROM…)
SELECT…FROMbJOINcON(…)
No comma!
WITH (non-recursive) Since SQL:1999
WITHa(c1,c2,c3)AS(SELECTc1,c2,c3FROM…),
b(c4,…)AS(SELECTc4,…FROMt1JOINaON(…)),
c(…)AS(SELECT…FROM…)
SELECT…FROMbJOINcON(…)
Main query
WITH (non-recursive) Since SQL:1999
CTEs are statement-scoped views:
WITHa(c1,c2,c3)AS(SELECTc1,c2,c3FROM…),
b(c4,…)AS(SELECTc4,…FROMt1JOINaON(…)),
c(…)AS(SELECT…FROM…)
SELECT…FROMbJOINcON(…)
Read top down
WITH (non-recursive) Since SQL:1999
‣ Literate SQL
Organize SQL code toimprove maintainability
‣ Assign column names
to tables produced by valuesor unnest.
‣ Overload tables (for testing)
with queries hide tablesof the same name.
Use-CasesWITH (non-recursive)
https://modern-sql.com/use-case/literate-sql
https://modern-sql.com/use-case/naming-unnamed-columns
https://modern-sql.com/use-case/unit-tests-on-transient-data
WITH are the "private methods" of SQL
WITH is a prefix to SELECT
WITH queries are only visible in the SELECT they precede
WITH in detail: https://modern-sql.com/feature/with
WITH (non-recursive) In a Nutshell
AvailabilityWITH (non-recursive)19
9920
0120
0320
0520
0720
0920
1120
1320
1520
17
5.1 10.2 MariaDB8.0 MySQL
8.4 PostgreSQL3.8.3[0] SQLite
7.0 DB2 LUW9iR2 Oracle
2005[1] SQL Server[0]Only for top-level SELECT statements[1]Only allowed at the very begin of a statement. E.g. WITH...INSERT...SELECT.
WITHRECURSIVE (Common Table Expressions)
(This page is intentionally left blank)
WITHRECURSIVE The Problem
CREATETABLEt(idNUMERICNOTNULL,parent_idNUMERIC,…PRIMARYKEY(id))
Coping with hierarchies in the Adjacency List Model[0]
WITHRECURSIVE The Problem
[0] Hierarchies implemented using a “parent id” — see “Joe Celko’s Trees and Hierarchies in SQL for Smarties”
SELECT*FROMtASd0LEFTJOINtASd1ON(d1.parent_id=d0.id)LEFTJOINtASd2ON(d2.parent_id=d1.id)
Coping with hierarchies in the Adjacency List Model[0]
WITHRECURSIVE The Problem
WHEREd0.id=?
[0] Hierarchies implemented using a “parent id” — see “Joe Celko’s Trees and Hierarchies in SQL for Smarties”
SELECT*FROMtASd0LEFTJOINtASd1ON(d1.parent_id=d0.id)LEFTJOINtASd2ON(d2.parent_id=d1.id)
Coping with hierarchies in the Adjacency List Model[0]
WITHRECURSIVE The Problem
WHEREd0.id=?
[0] Hierarchies implemented using a “parent id” — see “Joe Celko’s Trees and Hierarchies in SQL for Smarties”
SELECT*FROMtASd0LEFTJOINtASd1ON(d1.parent_id=d0.id)LEFTJOINtASd2ON(d2.parent_id=d1.id)
Coping with hierarchies in the Adjacency List Model[0]
WITHRECURSIVE The Problem
WHEREd0.id=?
[0] Hierarchies implemented using a “parent id” — see “Joe Celko’s Trees and Hierarchies in SQL for Smarties”
SELECT*FROMtASd0LEFTJOINtASd1ON(d1.parent_id=d0.id)LEFTJOINtASd2ON(d2.parent_id=d1.id)
WITHRECURSIVE Since SQL:1999
WHEREd0.id=?
WITHRECURSIVEd(id,parent,…)AS(SELECTid,parent,…FROMtblWHEREid=?UNIONALLSELECTid,parent,…FROMdJOINtblON(tbl.parent=d.id))SELECT*FROMsubtree
Recursive common table expressions may refer to themselves in a leg of a UNION[ALL]:
WITHRECURSIVEcte(n)AS(SELECT1UNIONALLSELECTn+1FROMcteWHEREn<3)SELECT*FROMcte
Keyword
Since SQL:1999WITHRECURSIVE
Recursive common table expressions may refer to themselves in a leg of a UNION[ALL]:
WITHRECURSIVEcte(n)AS(SELECT1UNIONALLSELECTn+1FROMcteWHEREn<3)SELECT*FROMcte
Column list mandatory here
Since SQL:1999WITHRECURSIVE
Recursive common table expressions may refer to themselves in a leg of a UNION[ALL]:
WITHRECURSIVEcte(n)AS(SELECT1UNIONALLSELECTn+1FROMcteWHEREn<3)SELECT*FROMcte
Executed first
Since SQL:1999WITHRECURSIVE
Recursive common table expressions may refer to themselves in a leg of a UNION[ALL]:
WITHRECURSIVEcte(n)AS(SELECT1UNIONALLSELECTn+1FROMcteWHEREn<3)SELECT*FROMcte
Result sent there
Since SQL:1999WITHRECURSIVE
Recursive common table expressions may refer to themselves in a leg of a UNION[ALL]:
WITHRECURSIVEcte(n)AS(SELECT1UNIONALLSELECTn+1FROMcteWHEREn<3)SELECT*FROMcte
Result visible twice
Since SQL:1999WITHRECURSIVE
Recursive common table expressions may refer to themselves in a leg of a UNION[ALL]:
WITHRECURSIVEcte(n)AS(SELECT1UNIONALLSELECTn+1FROMcteWHEREn<3)SELECT*FROMcte
n---123(3rows)
Once it becomes
part of the final result
Since SQL:1999WITHRECURSIVE
Recursive common table expressions may refer to themselves in a leg of a UNION[ALL]:
WITHRECURSIVEcte(n)AS(SELECT1UNIONALLSELECTn+1FROMcteWHEREn<3)SELECT*FROMcte
n---123(3rows)
Second leg of UNION is executed
Since SQL:1999WITHRECURSIVE
Recursive common table expressions may refer to themselves in a leg of a UNION[ALL]:
WITHRECURSIVEcte(n)AS(SELECT1UNIONALLSELECTn+1FROMcteWHEREn<3)SELECT*FROMcte
n---123(3rows)
Result sent there again
Since SQL:1999WITHRECURSIVE
Recursive common table expressions may refer to themselves in a leg of a UNION[ALL]:
WITHRECURSIVEcte(n)AS(SELECT1UNIONALLSELECTn+1FROMcteWHEREn<3)SELECT*FROMcte
n---123(3rows)
It's a loop!
Since SQL:1999WITHRECURSIVE
Recursive common table expressions may refer to themselves in a leg of a UNION[ALL]:
WITHRECURSIVEcte(n)AS(SELECT1UNIONALLSELECTn+1FROMcteWHEREn<3)SELECT*FROMcte
n---123(3rows)
It's a loop!
Since SQL:1999WITHRECURSIVE
Recursive common table expressions may refer to themselves in a leg of a UNION[ALL]:
WITHRECURSIVEcte(n)AS(SELECT1UNIONALLSELECTn+1FROMcteWHEREn<3)SELECT*FROMcte
n---123(3rows)
It's a loop!
Since SQL:1999WITHRECURSIVE
Recursive common table expressions may refer to themselves in a leg of a UNION[ALL]:
WITHRECURSIVEcte(n)AS(SELECT1UNIONALLSELECTn+1FROMcteWHEREn<3)SELECT*FROMcte
n---123(3rows)
n=3 doesn't matchLoop terminates
Since SQL:1999WITHRECURSIVE
AvailabilityWITHRECURSIVE19
9920
0120
0320
0520
0720
0920
1120
1320
1520
17
5.1 10.2 MariaDB8.0 MySQL
8.4 PostgreSQL3.8.3[0] SQLite
7.0 DB2 LUW11gR2 Oracle
2005 SQL Server[0]Only for top-level SELECT statements
GROUPINGSETS
Only one GROUPBY operation at a time:
GROUPINGSETS Before SQL:1999
SELECTyear,month,sum(revenue)FROMtblGROUPBYyear,month
Monthly revenue Yearly revenue
SELECTyear,sum(revenue)FROMtblGROUPBYyear
GROUPINGSETS Before SQL:1999SELECTyear,month,sum(revenue)FROMtblGROUPBYyear,month
SELECTyear,sum(revenue)FROMtblGROUPBYyear
GROUPINGSETS Before SQL:1999SELECTyear,month,sum(revenue)FROMtblGROUPBYyear,month
SELECTyear,sum(revenue)FROMtblGROUPBYyear
UNIONALL
,null
GROUPINGSETS Since SQL:1999SELECTyear,month,sum(revenue)FROMtblGROUPBYyear,month
SELECTyear,sum(revenue)FROMtblGROUPBYyear
UNIONALL
,null
SELECTyear,month,sum(revenue)FROMtblGROUPBYGROUPINGSETS((year,month),(year))
GROUPINGSETS are multiple GROUPBYs in one go
() (empty parenthesis) build a group over all rows
GROUPING (function) disambiguates the meaning of NULL(was the grouped data NULL or is this column not currently grouped?)
Permutations can be created using ROLLUP and CUBE(ROLLUP(a,b,c) = GROUPINGSETS((a,b,c),(a,b),(a),())
GROUPINGSETS In a Nutshell
GROUPINGSETS Availability19
9920
0120
0320
0520
0720
0920
1120
1320
1520
17
5.1[0] MariaDB5.0[1] MySQL
9.5 PostgreSQLSQLite
5 DB2 LUW9iR1 Oracle
2008 SQL Server[0]Only ROLLUP (properitery syntax).[1]Only ROLLUP (properitery syntax). GROUPING function since MySQL 8.0.
SQL:2003
FILTER
FILTER Before we start
In SQL, most aggregate functions* drop null arguments
prior to the aggregation.
*Exceptions: Aggregate functions that return structured data: array_agg, json_objectagg, json_arrayagg, xmlagg
See: http://modern-sql.com/concept/null#aggregates
SELECTYEAR,SUM(CASEWHENMONTH=1THENrevenueELSE0END)JAN,SUM(CASEWHENMONTH=2THENrevenueEND)FEB,…FROMsalesGROUPBYYEAR
FILTER The ProblemPivot table: Years on the Y axis, month on X:
SELECTYEAR,SUM(CASEWHENMONTH=1THENrevenueELSE0END)JAN,SUM(CASEWHENMONTH=2THENrevenueEND)FEB,…FROMsalesGROUPBYYEAR
FILTER The ProblemPivot table: Years on the Y axis, month on X:
Optional:ELSE null is default
SELECTYEAR,SUM(CASEWHENMONTH=1THENrevenueELSE0END)JAN,SUM(CASEWHENMONTH=2THENrevenueEND)FEB,…FROMsalesGROUPBYYEAR
FILTER The ProblemPivot table: Years on the Y axis, month on X:
SELECTYEAR,SUM(revenue)FILTER(WHEREMONTH=1)JAN,SUM(revenue)FILTER(WHEREMONTH=2)FEB,…FROMsalesGROUPBYYEAR;
FILTER Since SQL:2003SQL:2003 allows FILTER(WHERE…) after aggregates:
FILTER Since SQL:2003
Year
2016
2016
2016
2016
2016
Month
1
2
3
...
12
Revenue
1
23
345
...
1234
Year
2016
Jan
1
Feb
23
Mar
345
...
...
Dec
1234
SUM(…) FILTE
R(WHERE …)
SUM(…)
FILTER(WHER
E month=2)
SUM(rev
enue) FILTER(WHERE
month=3)
SUM(rev
enue) FILTER(WHERE mo
nth=…)
SUM(rev
enue) FILTER(WHERE month=
12)
Pivot in SQL1. Use GROUP BY
to combine rows2. Use FILTER to pick
rows per column
See: http://modern-sql.com/use-case/pivot
FILTER Availability19
9920
0120
0320
0520
0720
0920
1120
1320
1520
17
5.1 MariaDBMySQL
9.4 PostgreSQLSQLite
DB2 LUWOracleSQL Server
BOOLEANTests
Before we start
SQL uses a three-valued logic. Boolean values are either
true, false or unknown(=null).
See: http://modern-sql.com/concept/three-valued-logic
BOOLEANAggregates
BOOLEANTests
Similar to isnull, there are tests for each Boolean value(of which there are three: true, false, unknown/null)
IS[NOT][TRUE|FALSE|UNKNOWN]
Since SQL:2003
CREATETABLEprices(…valid_fromDATENOTNULL,valid_toDATE,--null:openend…CHECK(valid_from<valid_to),);
WHEREvalid_from<CURRENT_DATEAND(valid_to<=CURRENT_DATE)ISNOTFALSE
BOOLEANTests Since SQL:2003
CREATETABLEprices(…valid_fromDATENOTNULL,valid_toDATE,--null:openend…CHECK(valid_from<valid_to),);
WHEREvalid_from<CURRENT_DATEAND(valid_to<=CURRENT_DATE)ISNOTFALSE
BOOLEANTests Since SQL:2003
UNKNOWN ifVALID_TO is NULL
CREATETABLEprices(…valid_fromDATENOTNULL,valid_toDATE,--null:openend…CHECK(valid_from<valid_to),);
WHEREvalid_from<CURRENT_DATEAND(valid_to<=CURRENT_DATE)ISNOTFALSE
BOOLEANTests Since SQL:2003
UNKNOWN ifVALID_TO is NULL Takes TRUE and
UNKNOWN
BOOLEANTests Since SQL:200319
9920
0120
0320
0520
0720
0920
1120
1320
1520
17
5.1 MariaDB5.0.51a MySQL8.3 PostgreSQL3.5.7[0] SQLite
DB2 LUWOracleSQL Server
[0]No IS [NOT] UNKNOWN. Use IS [NOT] NULL instead.
OVER and
PARTITIONBY
OVER (PARTITION BY) The ProblemTwo distinct concepts could not be used independently:
‣Merge rows with the same key properties
‣ GROUPBY to specify key properties
‣ DISTINCT to use full row as key
‣ Aggregate data from related rows ‣ Requires GROUPBY to segregate the rows
‣ COUNT, SUM, AVG, MIN, MAX to aggregate grouped rows
SELECTc1,SUM(c2)totFROMtGROUPBYc1
OVER (PARTITION BY) The Problem
Yes ⇠
Mer
ge ro
ws ⇢
No
No ⇠ Aggregate ⇢ Yes
SELECTc1,c2FROMt
SELECTDISTINCTc1,c2FROMt
SELECTc1,c2FROMtJOIN()taON(t.c1=ta.c1)
SELECTc1,SUM(c2)totFROMtGROUPBYc1
,tot
SELECTc1,SUM(c2)totFROMtGROUPBYc1
OVER (PARTITION BY) The Problem
Yes ⇠
Mer
ge ro
ws ⇢
No
No ⇠ Aggregate ⇢ Yes
SELECTc1,c2FROMt
SELECTDISTINCTc1,c2FROMt
SELECTc1,c2FROMtJOIN()taON(t.c1=ta.c1)
SELECTc1,SUM(c2)totFROMtGROUPBYc1
,tot
SELECTc1,SUM(c2)totFROMtGROUPBYc1
OVER (PARTITION BY) Since SQL:2003
Yes ⇠
Mer
ge ro
ws ⇢
No
No ⇠ Aggregate ⇢ Yes
SELECTc1,c2FROMt
SELECTDISTINCTc1,c2FROMt
SELECTc1,c2FROMt
FROMt
,SUM(c2)OVER(PARTITIONBYc1)
SELECTdep,salary,SUM(salary)OVER()FROMemp
dep salary ts1 1000 600022 1000 600022 1000 6000333 1000 6000333 1000 6000333 1000 6000
OVER (PARTITION BY) How it works
SELECTdep,salary,SUM(salary)OVER()FROMemp
dep salary ts1 1000 600022 1000 600022 1000 6000333 1000 6000333 1000 6000333 1000 6000
OVER (PARTITION BY) How it works
SELECTdep,salary,SUM(salary)OVER()FROMemp
dep salary ts1 1000 600022 1000 600022 1000 6000333 1000 6000333 1000 6000333 1000 6000
OVER (PARTITION BY) How it works
SELECTdep,salary,SUM(salary)OVER()FROMemp
dep salary ts1 1000 600022 1000 600022 1000 6000333 1000 6000333 1000 6000333 1000 6000
OVER (PARTITION BY) How it works
SELECTdep,salary,SUM(salary)OVER()FROMemp
dep salary ts1 1000 600022 1000 600022 1000 6000333 1000 6000333 1000 6000333 1000 6000
OVER (PARTITION BY) How it works
SELECTdep,salary,SUM(salary)OVER()FROMemp
dep salary ts1 1000 600022 1000 600022 1000 6000333 1000 6000333 1000 6000333 1000 6000
OVER (PARTITION BY) How it works
SELECTdep,salary,SUM(salary)OVER()FROMemp
dep salary ts1 1000 100022 1000 200022 1000 2000333 1000 3000333 1000 3000333 1000 3000
OVER (PARTITION BY) How it works
)PARTITIONBYdep
SELECTdep,salary,SUM(salary)OVER()FROMemp
dep salary ts1 1000 100022 1000 200022 1000 2000333 1000 3000333 1000 3000333 1000 3000
OVER (PARTITION BY) How it works
)PARTITIONBYdep
SELECTdep,salary,SUM(salary)OVER()FROMemp
dep salary ts1 1000 100022 1000 200022 1000 2000333 1000 3000333 1000 3000333 1000 3000
OVER (PARTITION BY) How it works
)PARTITIONBYdep
OVER and
ORDERBY(Framing & Ranking)
acnt id value balance
1 1 +10 +10
22 2 +20 +30
22 3 -10 +20
333 4 +50 +70
333 5 -30 +40
333 6 -20 +20
OVER (ORDER BY) The Problem
SELECTid,value,FROMtransactionst
acnt id value balance
1 1 +10 +10
22 2 +20 +30
22 3 -10 +20
333 4 +50 +70
333 5 -30 +40
333 6 -20 +20
OVER (ORDER BY) The Problem
SELECTid,value,FROMtransactionst
acnt id value balance
1 1 +10 +10
22 2 +20 +30
22 3 -10 +20
333 4 +50 +70
333 5 -30 +40
333 6 -20 +20
OVER (ORDER BY) The Problem
SELECTid,value,
(SELECTSUM(value)FROMtransactionst2WHEREt2.id<=t.id)
FROMtransactionst
Range segregation (<=)not possible with
GROUP BY orPARTITION BY
OVER (ORDER BY) Since SQL:2003
SELECTid,value,
FROMtransactionst
SUM(value)OVER(
)
acnt id value balance
1 1 +10 +10
22 2 +20 +30
22 3 -10 +20
333 4 +50 +70
333 5 -30 +40
333 6 -20 +20
OVER (ORDER BY) Since SQL:2003
SELECTid,value,
FROMtransactionst
SUM(value)OVER(
)
acnt id value balance
1 1 +10 +10
22 2 +20 +30
22 3 -10 +20
333 4 +50 +70
333 5 -30 +40
333 6 -20 +20
ORDERBYid
OVER (ORDER BY) Since SQL:2003
SELECTid,value,
FROMtransactionst
SUM(value)OVER(
)
acnt id value balance
1 1 +10 +10
22 2 +20 +30
22 3 -10 +20
333 4 +50 +70
333 5 -30 +40
333 6 -20 +20
ORDERBYidROWSBETWEENUNBOUNDEDPRECEDING
OVER (ORDER BY) Since SQL:2003
SELECTid,value,
FROMtransactionst
SUM(value)OVER(
)
acnt id value balance
1 1 +10 +10
22 2 +20 +30
22 3 -10 +20
333 4 +50 +70
333 5 -30 +40
333 6 -20 +20
ORDERBYidROWSBETWEENUNBOUNDEDPRECEDINGANDCURRENTROW
OVER (ORDER BY) Since SQL:2003
SELECTid,value,
FROMtransactionst
SUM(value)OVER(
)
acnt id value balance
1 1 +10 +10
22 2 +20 +30
22 3 -10 +20
333 4 +50 +70
333 5 -30 +40
333 6 -20 +20
ORDERBYidROWSBETWEENUNBOUNDEDPRECEDINGANDCURRENTROW
OVER (ORDER BY) Since SQL:2003
SELECTid,value,
FROMtransactionst
SUM(value)OVER(
)
acnt id value balance
1 1 +10 +10
22 2 +20 +30
22 3 -10 +20
333 4 +50 +70
333 5 -30 +40
333 6 -20 +20
ORDERBYidROWSBETWEENUNBOUNDEDPRECEDINGANDCURRENTROW
OVER (ORDER BY) Since SQL:2003
SELECTid,value,
FROMtransactionst
SUM(value)OVER(
)
acnt id value balance
1 1 +10 +10
22 2 +20 +30
22 3 -10 +20
333 4 +50 +70
333 5 -30 +40
333 6 -20 +20
ORDERBYidROWSBETWEENUNBOUNDEDPRECEDINGANDCURRENTROW
OVER (ORDER BY) Since SQL:2003
SELECTid,value,
FROMtransactionst
SUM(value)OVER(
)
acnt id value balance
1 1 +10 +10
22 2 +20 +30
22 3 -10 +20
333 4 +50 +70
333 5 -30 +40
333 6 -20 +20
ORDERBYidROWSBETWEENUNBOUNDEDPRECEDINGANDCURRENTROW
OVER (ORDER BY) Since SQL:2003
SELECTid,value,
FROMtransactionst
SUM(value)OVER(
)
acnt id value balance
1 1 +10 +10
22 2 +20 +30
22 3 -10 +20
333 4 +50 +70
333 5 -30 +40
333 6 -20 +20
ORDERBYidROWSBETWEENUNBOUNDEDPRECEDINGANDCURRENTROW
OVER (ORDER BY) Since SQL:2003
SELECTid,value,
FROMtransactionst
SUM(value)OVER(
)
acnt id value balance
1 1 +10 +10
22 2 +20 +30
22 3 -10 +20
333 4 +50 +70
333 5 -30 +40
333 6 -20 +20
ORDERBYidROWSBETWEENUNBOUNDEDPRECEDINGANDCURRENTROW
OVER (ORDER BY) Since SQL:2003
SELECTid,value,
FROMtransactionst
SUM(value)OVER(
)
acnt id value balance
1 1 +10 +10
22 2 +20 +20
22 3 -10 +10
333 4 +50 +50
333 5 -30 +20
333 6 -20 .0
ORDERBYidROWSBETWEENUNBOUNDEDPRECEDINGANDCURRENTROW
PARTITIONBYacnt
OVER (ORDER BY) Since SQL:2003With OVER(ORDERBYn) a new type of functions make sense:
n ROW_NUMBER RANK DENSE_RANK PERCENT_RANK CUME_DIST1 1 1 1 0 0.252 2 2 2 0.33… 0.753 3 2 2 0.33… 0.754 4 4 3 1 1
‣ Aggregates without GROUPBY
‣ Running totals, moving averages
‣ Ranking‣ Top-N per Group
‣ Avoiding self-joins
[… many more …]
Use Cases
SELECT*FROM(SELECTROW_NUMBER()OVER(PARTITIONBY…ORDERBY…)rn,t.*FROMt)numbered_tWHERErn<=3
AVG(…)OVER(ORDERBY…ROWSBETWEEN3PRECEDINGAND3FOLLOWING)moving_avg
OVER (SQL:2003)
OVER may follow any aggregate function
OVER defines which rows are visible at each row
OVER() makes all rows visible at every row
OVER(PARTITIONBY …) segregates like GROUPBY
OVER(ORDERBY…BETWEEN) segregates using <, >
In a NutshellOVER (SQL:2003)
OVER (SQL:2003) Availability19
9920
0120
0320
0520
0720
0920
1120
1320
1520
17
5.1 10.2 MariaDB8.0 MySQL
8.4 PostgreSQLSQLite
7.0 DB2 LUW8i Oracle
2005 SQL Server
OVER (SQL:2003) Availability19
9920
0120
0320
0520
0720
0920
1120
1320
1520
17
5.1 10.2 MariaDB8.0 MySQL
8.4 PostgreSQLSQLite
7.0 DB2 LUW8i Oracle
2005 SQL Server
Hive
ImpalaSpark
NuoDB
SQL:2011
OVER
WITHnumbered_tAS(SELECT*)
SELECTcurr.*,curr.balance-COALESCE(prev.balance,0)FROMnumbered_tcurrLEFTJOINnumbered_tprevON(curr.rn=prev.rn+1)
OVER (SQL:2011) The ProblemDirect access of other rows of the same window is not possible.
(E.g., calculate the difference to the previous rows)
currbalance … rn
50 … 190 … 270 … 330 … 4
FROMt
WITHnumbered_tAS(SELECT*)
SELECTcurr.*,curr.balance-COALESCE(prev.balance,0)FROMnumbered_tcurrLEFTJOINnumbered_tprevON(curr.rn=prev.rn+1)
OVER (SQL:2011) The ProblemDirect access of other rows of the same window is not possible.
(E.g., calculate the difference to the previous rows)
currbalance … rn
50 … 190 … 270 … 330 … 4
FROMt,ROW_NUMBER()OVER(ORDERBYx)rn
WITHnumbered_tAS(SELECT*)
SELECTcurr.*,curr.balance-COALESCE(prev.balance,0)FROMnumbered_tcurrLEFTJOINnumbered_tprevON(curr.rn=prev.rn+1)
OVER (SQL:2011) The ProblemDirect access of other rows of the same window is not possible.
(E.g., calculate the difference to the previous rows)
currbalance … rn
50 … 190 … 270 … 330 … 4
FROMt,ROW_NUMBER()OVER(ORDERBYx)rn
WITHnumbered_tAS(SELECT*)
SELECTcurr.*,curr.balance-COALESCE(prev.balance,0)FROMnumbered_tcurrLEFTJOINnumbered_tprevON(curr.rn=prev.rn+1)
OVER (SQL:2011) The ProblemDirect access of other rows of the same window is not possible.
(E.g., calculate the difference to the previous rows)
currbalance … rn
50 … 190 … 270 … 330 … 4
FROMt,ROW_NUMBER()OVER(ORDERBYx)rn
WITHnumbered_tAS(SELECT*)
SELECTcurr.*,curr.balance-COALESCE(prev.balance,0)FROMnumbered_tcurrLEFTJOINnumbered_tprevON(curr.rn=prev.rn+1)
OVER (SQL:2011) The ProblemDirect access of other rows of the same window is not possible.
(E.g., calculate the difference to the previous rows)
currbalance … rn
50 … 190 … 270 … 330 … 4
FROMt,ROW_NUMBER()OVER(ORDERBYx)rn
prevbalance … rn
50 … 190 … 270 … 330 … 4
WITHnumbered_tAS(SELECT*)
SELECTcurr.*,curr.balance-COALESCE(prev.balance,0)FROMnumbered_tcurrLEFTJOINnumbered_tprevON(curr.rn=prev.rn+1)
OVER (SQL:2011) The ProblemDirect access of other rows of the same window is not possible.
(E.g., calculate the difference to the previous rows)
currbalance … rn
50 … 190 … 270 … 330 … 4
FROMt,ROW_NUMBER()OVER(ORDERBYx)rn
prevbalance … rn
50 … 190 … 270 … 330 … 4
WITHnumbered_tAS(SELECT*)
SELECTcurr.*,curr.balance-COALESCE(prev.balance,0)FROMnumbered_tcurrLEFTJOINnumbered_tprevON(curr.rn=prev.rn+1)
OVER (SQL:2011) The ProblemDirect access of other rows of the same window is not possible.
(E.g., calculate the difference to the previous rows)
currbalance … rn
50 … 190 … 270 … 330 … 4
FROMt,ROW_NUMBER()OVER(ORDERBYx)rn
prevbalance … rn
50 … 190 … 270 … 330 … 4
+50+40-20-40
SELECT*,balance-COALESCE(LAG(balance)OVER(ORDERBYx),0)FROMt
Available functions:LEAD/LAGFIRST_VALUE/LAST_VALUENTH_VALUE(col,n)FROMFIRST/LASTRESPECT/IGNORENULLS
OVER (SQL:2011) Since SQL:2011SQL:2011 introduced LEAD, LAG, NTH_VALUE, … for that:
OVER (LEAD, LAG, …) Since SQL:201119
9920
0120
0320
0520
0720
0920
1120
1320
1520
17
5.1 10.2[0] MariaDB8.0[0] MySQL
8.4[0] PostgreSQLSQLite
9.5[1] 11.1 DB2 LUW8i[1] 11gR2 Oracle
2012[1] SQL Server[0]No IGNORENULLS and FROMLAST[1]No NTH_VALUE
System Versioning (Time Traveling)
INSERTUPDATEDELETE
are DESTRUCTIVE
System Versioning The Problem
ID Data start_ts end_ts1 X 10:00:00
UPDATE...SETDATA='Y'...
ID Data start_ts end_ts1 X 10:00:00 11:00:001 Y 11:00:00
DELETE...WHEREID=1
INSERT...(ID,DATA)VALUES(1,'X')
System Versioning Since SQL:2011
ID Data start_ts end_ts1 X 10:00:00
UPDATE...SETDATA='Y'...
ID Data start_ts end_ts1 X 10:00:00 11:00:001 Y 11:00:00
DELETE...WHEREID=1
ID Data start_ts end_ts1 X 10:00:00 11:00:001 Y 11:00:00 12:00:00
INSERT...(ID,DATA)VALUES(1,'X')
System Versioning Since SQL:2011
Although multiple versions exist, only the “current” one is visible per default.
After 12:00:00, SELECT*FROMt doesn’t return anything anymore.
ID Data start_ts end_ts1 X 10:00:00 11:00:001 Y 11:00:00 12:00:00
System Versioning Since SQL:2011
ID Data start_ts end_ts1 X 10:00:00 11:00:001 Y 11:00:00 12:00:00
With FOR…ASOF you can query anything you like: SELECT*FROMtFORSYSTEM_TIMEASOFTIMESTAMP'2015-04-0210:30:00'
ID Data start_ts end_ts
1 X 10:00:00 11:00:00
System Versioning Since SQL:2011
1999
2001
2003
2005
2007
2009
2011
2013
2015
2017
5.1 MariaDB[0]
MySQLPostgreSQLSQLite
10.1[1] DB2 LUW10gR1[2] Oracle
2016 SQL Server[0]Available in MariaDB 10.3 beta.[1]Third column required (tx id), history table required.[2]Functionality available using Flashback
System Versioning Since SQL:2011
SQL:2016 (released: 2016-12-15)
New in SQL:2016 JSON
ROW PATTERN MATCHING https://www.slideshare.net/MarkusWinand/row-pattern-matching-in-sql2016
DATE FORMAT
POLYMORPHIC TABLE FUNCTIONS ➔ https://modern-sql.com/blog/2017-06/whats-new-in-sql-2016
SQL has evolved beyond the relational idea.
Modern SQL? @MarkusWinand
SQL has evolved beyond the relational idea.
If you are using SQL like 25 years ago,you are doing it wrong!
Modern SQL? @MarkusWinand
SQL has evolved beyond the relational idea.
If you are using SQL like 25 years ago,you are doing it wrong!
Don’t think SQL cannot do this or that.Act professionally: check it out before complaining.
Modern SQL? @MarkusWinand
Act professionally: check it out
SQL has evolved beyond the relational idea.
If you are using SQL like 25 years ago,you are doing it wrong!
Don’t think SQL cannot do this or that.Act professionally: check it out before complaining.
Modern SQL? @MarkusWinand
Act professionally: check it out
About @MarkusWinand
‣Training for Developers ‣ SQL Performance (Indexing) ‣ Modern SQL ‣ On-Site or Online
‣SQL Tuning ‣ Index-Redesign ‣ Query Improvements ‣ On-Site or Online
https://winand.at/
About @MarkusWinand€0,-
€10-30‣Training for Developers ‣ SQL Performance (Indexing) ‣ Modern SQL ‣ On-Site or Online
‣SQL Tuning ‣ Index-Redesign ‣ Query Improvements ‣ On-Site or Online
sql-performance-explained.com
About @MarkusWinand@ModernSQL
https://modern-sql.com‣Training for Developers ‣ SQL Performance (Indexing) ‣ Modern SQL ‣ On-Site or Online
‣SQL Tuning ‣ Index-Redesign ‣ Query Improvements ‣ On-Site or Online