Row Pattern Matching in SQL:2016

Row Pattern Matching@MarkusWinand

Image: 72hoursamericanpower.com

Row Pattern Matching Availability19

99

2001

2003

2005

2007

2009

2011

2013

2015

MariaDBMySQLPostgreSQLSQLite

DB2 LUW12cR1 Oracle

SQL Server

Grouping Consecutive Events

Time

30 minutes

Example: Logfile

Grouping Consecutive Events

Example: Logfile

Time

30 minutes

Session 1 Session 2 Session 3

Session 4

Example problems:

‣ Count sessions

‣ Average session duration

Two approaches:

‣ Start-of-group tagging

‣ Row pattern matching

SELECTCOUNT(grp_start)groupsFROM(SELECTCASEWHENts>LAG(ts,1,DATE'1900-01-01')OVER(ORDERBYts)+INTERVAL'30'minuteTHEN1ENDgrp_startFROMlog)T

Consecutive Events: Counting Start-of-group tagging

Time

30 minutes

SELECTCOUNT(grp_start)groupsFROM(SELECTCASEWHENts>LAG(ts,1,DATE'1900-01-01')OVER(ORDERBYts)+INTERVAL'30'minuteTHEN1ENDgrp_startFROMlog)T

Consecutive Events: Counting Start-of-group tagging

Time

30 minutes

count thenon-NULL

values

SELECTCOUNT(*)sessionsFROMlogMATCH_RECOGNIZE(ORDERBYtsPATTERN(new)DEFINEnewASts>COALESCE(PREV(ts),DATE'1900-01-01')+INTERVAL'30'minute)t

Consecutive Events: Counting Row Pattern Matching

Time

30 minutes



Time

30 minutes

row pattern variable



Time

30 minutes

match only “new”

rows



Time

30 minutes

countrows

SELECTCOUNT(*)sessions,AVG(duration)avg_durationFROMlogMATCH_RECOGNIZE(ORDERBYtsMEASURESLAST(ts)-FIRST(ts)ASdurationONEROWPERMATCHPATTERN(newcont*)DEFINEcontASts<PREV(ts)+INTERVAL'30'minute)t

Row Pattern MatchingConsecutive Events: Statistics

Time

30 minutes

definecontinuation

Oracle doesn’t support avg on intervals — query doesn’t work as shown



Time

30 minutes

undefinedpattern variable: matches any row




Time

30 minutes

any numberof “cont”

rows




Time

30 minutes

Very muchlike GROUP BY




Time

30 minutes

Very muchlike SELECT




Time

30 minutes


Consecutive Events: Statistics Start-of-group tagging

Time

30 minutes

Now, let’s try using window functions

SELECTcount(*)sessions,avg(duration)avg_durationFROM(SELECTMAX(ts)-MIN(ts)durationFROM(SELECTts,COUNT(grp_start)OVER(ORDERBYts)session_noFROM(SELECTts,CASEWHENts>=LAG(ts,1,DATE’1900-01-1')OVER(ORDERBYts)+INTERVAL'30'minuteTHEN1ENDgrp_startFROMlog)tagged)numberedGROUPBYsession_no)grouped


Time

30 minutes

Start-of-group tags



Time

30 minutes

number sessions

2222 2 33 3 44 42 3 41



Time

30 minutes 2222 2 33 3 44 42 3 41



Time

30 minutes

4 Levels: 2 with window functions

2 for grouping

What about performance?

2222 2 33 3 44 42 3 41

Tolerating Gaps

Example: Comments (new vs. read)

Tolerating Gaps

Example: Comments (new vs. read)

Show comments which…

‣…are new or

‣…between two new ones (show the comment instead of a “load more” button)

Two approaches:

‣ Start-of-group tagging

‣ Row pattern matching

Comments

new comment read comment

SELECTid,markerFROMmsgMATCH_RECOGNIZE(ORDERBYidALLROWSPERMATCHPATTERN(new+(readnew+)*)DEFINEnewAS(marker='X'))TORDERBYid

Tolerating Gaps Row Pattern Matching

Comments

Start with oneor more NEWcomment(s)



Comments


Doesn’t match



Comments


Two rows match “new+”



Comments

Match exactlyone row (any)



Comments



Comments

Repeat groupany numberof times



Comments

Repeat groupany numberof times



CommentsFirst match



CommentsSecondmatch



Comments

Tolerating Gaps (also first/last) Row Pattern Matching

CommentsWhat about

this?

SELECTid,markerFROMmsgMATCH_RECOGNIZE(ORDERBYidALLROWSPERMATCHPATTERN((^read)?new+(readnew+)*(read$)?)DEFINEnewAS(marker='X'))TORDERBYthread_id,id


CommentsWhat about

this?



Comments

Match“read” at the very

beginning



Comments

Optionally match“read” at the very

beginning



Comments

Tolerating Gaps lead & lag

Comments

Now, let’s try using window functions

SELECTt.*FROM(SELECTmsg.*,LAG(marker,1,'X')OVER(ORDERBYid)prev_marker,LEAD(marker,1,'X')OVER(ORDERBYid)next_markerFROMmsg)tWHEREmarker='X'OR(prev_marker='X'andnext_marker=‘X')ORDERBYid

Tolerating Gaps lead & lag

Comments

I don't care what anything was designed to do, I care about what it can do.

—Apollo 13, Universal Pictures

Tolerating Gaps (with grouped gaps) Row Pattern Matching

Comments


CommentsLoad 2 more Load 9 more



Tell me how many rows you skipped in between

SELECTid,marker,gap_lengthFROMmsgMATCH_RECOGNIZE(ORDERBYidMEASURESFINALCOUNT(more.*)ASgap_lengthALLROWSPERMATCHPATTERN((^read)?new+(readnew+)*(read$)?|more{-more*-})DEFINEnewAS(marker='X'),moreAS(marker!='X'))TORDERBYid






AlternativeMatch, but don’t return




Considerall rows



CommentsLoad 2 more Load 9 moreOnly rows

matched to the pattern variable

“more”

SELECTid,marker,CASEWHENmarker!='X'ANDgap_length>2THENgap_lengthENDgap_lengthFROM(SELECTt2.*,COUNT(CASEWHENmarker!='X'THEN1END)OVER(PARTITIONBYnew_counter)gap_lengthFROM(SELECTmsg.*,COUNT(CASEWHENmarker='X'THEN1END)OVER(ORDERBYid)new_counter,LAG(marker,1,'X')OVER(ORDERBYid)prev_markerFROMmsg)t2)t3WHEREmarker='X'ORgap_length=1ORprev_marker='X'ORDERBYid

Start-of-group taggingTolerating Gaps (with grouped gaps)

Comments

Top-N Per Group

Example: List 3 most recent comments per topic

Top-N Per Group


Time

Topic 3Topic 2Topic 1

Top-N Per Group


Three approaches:

‣ lateral sub-query (requires specific indexing)

‣ Row pattern matching (requires 12c)

‣ row_number() window function

Time


SELECT*FROMtMATCH_RECOGNIZE(PARTITIONBYtopicORDERBYvalMEASURESRUNNINGCOUNT(*)ASrnALLROWSPERMATCHPATTERN(^a{1,3})DEFINEaAS1=1)

Time


Top-N Per Group

per “topic” processing


Time


Top-N Per Group

Consider rows up till current

row


Time


Top-N Per Group

1, 2, or 3 times


Time


Top-N Per Group

DEFINE is non-optional: Use dummy


SELECT*FROM(SELECTt.*,ROW_NUMBER()OVER(PARTITIONBYtopicORDERBYval)rnFROMt)tWHERErn<=3

Time


Top-N Per Group

SELECT*FROMtMATCH_RECOGNIZE(PARTITIONBYtopicORDERBYvalMEASURESRUNNINGCOUNT(*)ASrnALLROWSPERMATCHPATTERN(^a+)DEFINEaAScount(*)<=3)

SELECT*FROM(SELECTt.*,ROW_NUMBER()OVER(PARTITIONBYtopicORDERBYval)rnFROMt)tWHERErn<=3

Time


Top-N Per Group

Always RUNNING semantic

Time Intervals (non-overlapping)

Example: Bookings are stored as [begin; end[ intervals


Example: Bookings are stored as [begin; end[ intervals

Two problems:

‣ Find free time-slots

‣ Close free time-slots

Time

Busy Busy Busy

Time

SELECT*FROMreservationsMATCH_RECOGNIZE(ORDERBYbeginMEASURESa.endASbegin,b.beginASendONEROWPERMATCHAFTERMATCHSKIPTObPATTERN(ab)DEFINEbASa.end<begin)


Time


a bTime Intervals (non-overlapping)

Time



Time





Time



Time



Time

Default is tocontinue AFTERlast matched row



Time



Time

a b



a bTime



Time



Time



Time

SELECT*FROM(SELECTendbegin,LEAD(begin)OVER(ORDERBYbegin)endFROMreservations)WHEREbegin<end

Row Pattern Matching

Time

Time Intervals (close gaps)

SELECTbbegin,eend,typeFROMreservationsMATCH_RECOGNIZE(ORDERBYbeginMEASURESCASEWHENfree.beginISNULLTHENbusy.beginELSEbusy.endENDASb,COALESCE(free.begin,busy.end)ASe,CLASSIFIER()astypeALLROWSPERMATCHAFTERMATCHSKIPTONEXTROWPATTERN(busyfree?)DEFINEfreeASbegin>PREV(end))


Time


Always matchone row. Second only

if there is a gap

Busy Free



Time


Busy Free



Time


Busy FreeIf it is not a

“free” row, passrow through


Row Pattern MatchingTime Intervals (close gaps)

Busy FreeTime



Busy FreeTime

Free



Busy FreeTime

Free



BusyTime

Free



BusyTime

Busy FreeFree



Busy Busy FreeFreeTime

Free



Busy BusyFreeTime

Free Busy

SELECTbegin,end,typeFROM(SELECTendbegin,LEAD(begin)OVER(ORDERBYbegin)end,'FREE'typeFROMreservations)WHEREbegin<endUNIONALLSELECTbegin,end,'BUSY'typeFROMreservations

Busy BusyFreeTime

Free BusyTime Intervals (close gaps) Window function

Endless possibilitesRow Pattern Matching

GROUPBY➡ONEROWPERMATCH

OVER()➡ALLROWSPERMATCH,FINAL,RUNNINGHAVING,WHERE➡PATTERN (unmatched, suppressed {-…-})

Mixing GROUPBY and OVER()➡ALLROWSPERMATCH + all-but-one rows suppressed

Data-driven match length ➡ SUM, COUNT, … in DEFINE

Duplicating rows (to some extend)➡ ALLROWSPERMATCH + AFTERMATCHSKIPTO…

What if I told you, you can also

find patterns?

Not/Barely covered in this presentationRow Pattern Matching

‣ Reluctant (non-greedy) matching

‣ SHOW/OMITEMPTYMATCHESWITHUNMATCHEDROWS

‣ SUBSET (define pattern-vars for use in MEASURES,DEFINE and AFTERMATCHSKIPTO)

‣ PREV, NEXT, FIRST, LAST (with some nesting!)

‣MATCH_NUMBER()

Obstacles and Issues (as of 12r1)Row Pattern Matching

‣ JDBC !!!!!Tokens ?, {, } have special meaning in JDBC.You have to escape them using {\...\}https://docs.oracle.com/database/121/JJDBC/apxref.htm#CHECHCJH

‣ ORA-62513: Quantified subpatterns that can have empty matches are not yet supported

PATTERN(x(a*b*)+y)

‣ ORA-62512: This aggregate is not yet supported in MATCH_RECOGNIZE clause.(only COUNT, SUM, AVG, MIN, and MAX)

About @MarkusWinand

‣Training for Developers ‣ SQL Performance (Indexing) ‣ Modern SQL ‣ On-Site or Online

‣SQL Tuning ‣ Index-Redesign ‣ Query Improvements ‣ On-Site or Online

http://winand.at/

About @MarkusWinand

€0,-€10-30

‣Training for Developers ‣ SQL Performance (Indexing) ‣ Modern SQL ‣ On-Site or Online


sql-performance-explained.com

About @MarkusWinand@ModernSQL

http://modern-sql.com‣Training for Developers ‣ SQL Performance (Indexing) ‣ Modern SQL ‣ On-Site or Online


Data & Analytics

Row Pattern Matching in SQL:2016