About MySQL Development (Optimizer Mainly)- WITH RECURSIVE and MySQL

lundi 18 novembre 2013

avec Google Friend Connect

Membres (9)

Vous êtes déjà membre ? Connexion

Membres

► 2015 (1)

► 2014 (2)

▼ 2013 (2)

▼ novembre (1)

WITH RECURSIVE and MySQL

► février (1)

Archives du blog

Más Siguiente blog» Crear un blog AccederEste sitio emplea cookies de Google para prestar sus servicios, para personalizar anuncios y para analizar el tráfico. Google recibe información

sobre tu uso de este sitio web. Si utilizas este sitio web, se sobreentiende que aceptas el uso de cookies. Más información Entendido

About MySQL development (Optimizer mainly): WITH RECURSIVE and MySQL http://guilhembichot.blogspot.com.es/2013/11/with-recursive-and-mysql.html

1 de 13 31/07/2015 4:01

(SELECT YEAR, SUM(SALES) AS S FROM T1 GROUP BY YEAR) AS D2

WHERE D1.YEAR = D2.YEAR-1;

Both derived tables are based on the same subquery text, but usually a DBMS is not

smart enough to recognize it. Thus, it will evaluate "SELECT YEAR, SUM(SALES)...

GROUP BY YEAR" twice! A first time to fill D1, a second time to fill D2. This limitation is

sometimes stated as "it's not possible to refer to a derived table twice in the same

query".

Such double evaluation can lead to a serious performance problem. Using WITH, this

limitation does not exist, and the following statement evaluates the subquery only once:

WITH D AS (SELECT YEAR, SUM(SALES) AS S FROM T1 GROUP BY YEAR)

SELECT D1.YEAR, (CASE WHEN D1.S>D2.S THEN 'INCREASE' ELSE 'DECREASE' END) AS TREND

FROM

D AS D1,

D AS D2


This already demonstrates one benefit of WITH.

In MySQL, WITH is not yet supported. But it can be emulated with a view:

CREATE VIEW D AS (SELECT YEAR, SUM(SALES) AS S FROM T1 GROUP BY YEAR);

SELECT D1.YEAR, (CASE WHEN D1.S>D2.S THEN 'INCREASE' ELSE 'DECREASE' END) AS TREND

FROM

D AS D1,

D AS D2


DROP VIEW D;

Instead of a view, I could as well create D as a normal table. But not as a temporary table,

because in MySQL a temporary table cannot be referred twice in the same query, as

mentioned in the manual.

Guilhem Bichot

Hi. I'm a developer of the

MySQL open-source

database server, at Oracle

Corporation. More about me:

http://www.linkedin.com/profile

/view?id=14387841 .

Afficher mon profil complet

Qui êtes-vous ?

Slides from Percona Live and airbnb TechTalks - 21/4/2015 - Øystein

MySQL Webinar: Analyze & Tune Queriesfor BetterPerformance - 25/2/2015 - Øystein

MySQL Webinar: Analyze & Tune Queriesfor BetterPerformance - 18/2/2015 - Øystein

When ONLY_FULL_GROUP_BY Won’tSee the Query IsDeterministic… - 19/1/2015 - GuilhemBichot

MySQL 5.7: only_full_group_by Improved,Recognizing Functional Dependencies,Enabled by Default! - 10/12/2014 - GuilhemBichot

MySQL Optimizer Blogs

Este sitio emplea cookies de Google para prestar sus servicios, para personalizar anuncios y para analizar el tráfico. Google recibe información



2 de 13 31/07/2015 4:01

According to the SQL standard, to use the recursive form, you should write WITH

RECURSIVE. However, looking at some other DBMSs, they seem to not require the

RECURSIVE word.

WITH RECURSIVE is a powerful construct. For example, it can do the same job as

Oracle's CONNECT BY clause (you can check out some example conversions between

both constructs).

Let's walk through an example, to understand what WITH RECURSIVE does.

Assume you have a table of employees (this is a very classical example of WITH

RECURSIVE):

CREATE TABLE EMPLOYEES (

ID INT PRIMARY KEY,

NAME VARCHAR(100),

MANAGER_ID INT,

INDEX (MANAGER_ID),

FOREIGN KEY (MANAGER_ID) REFERENCES EMPLOYEES(ID)

);

INSERT INTO EMPLOYEES VALUES

(333, "Yasmina", NULL),

(198, "John", 333),

(29, "Pedro", 198),

(4610, "Sarah", 29),

(72, "Pierre", 29),

(692, "Tarek", 333);

In other words, Yasmina is CEO, John and Tarek report to her. Pedro reports to John,

Sarah and Pierre report to Pedro.

In a big company, they would be thousands of rows in this table.

Now, let's say that you would like to know, for each employee: "how many people are,

directly and indirectly, reporting to him/her"? Here is how I would do it. First, I would




3 de 13 31/07/2015 4:01

SELECT ID, NAME, MANAGER_ID, 0 AS REPORTS

FROM EMPLOYEES

WHERE ID NOT IN (SELECT MANAGER_ID FROM EMPLOYEES WHERE MANAGER_ID IS NOT NULL);

Then I would insert the results into a new table named EMPLOYEES_EXTENDED;

EXTENDED stands for "extended with more information", the new information being

the fourth column named REPORTS: it is a count of people who are reporting directly or

indirectly to the employee. Because we have listed people who are not managers, they

have a value of 0 in the REPORTS column.

Then, we can produce the rows for "first level" managers (the direct managers of

non-managers):

SELECT M.ID, M.NAME, M.MANAGER_ID, SUM(1+E.REPORTS) AS REPORTS

FROM EMPLOYEES M JOIN EMPLOYEES_EXTENDED E ON M.ID=E.MANAGER_ID

GROUP BY M.ID, M.NAME, M.MANAGER_ID;

Explanation: for a row of M (that is, for an employee), the JOIN will produce zero or

more rows, one per non-manager directly reporting to the employee.

Each such non-manager contributes to the value of REPORTS for his manager, through

two numbers: 1 (the non-manager himself), and the number of direct/indirect reports of

the non-manager (i.e. the value of REPORTS for the non-manager).

Then I would empty EMPLOYEES_EXTENDED, and fill it with the rows produced just

above, which describe the first level managers.

Then the same query should be run again, and it would produce information about the

"second level" managers. And so on.

Finally, at one point Yasmina will be the only row of EMPLOYEES_EXTENDED, and

when we run the above SELECT again, the JOIN will produce no rows, because

E.MANAGER_ID will be NULL (she's the CEO). We are done.

It's time for a recap: EMPLOYEES_EXTENDED has been a kind of "temporary buffer",

which has successively held non-managers, first level managers, second level managers,




4 de 13 31/07/2015 4:01

member" or "the seed". The SELECT query which moves from one step of recursion to

the next one, is the "recursive member". The complete statement looks like this:

WITH RECURSIVE

# The temporary buffer, also used as UNION result:

EMPLOYEES_EXTENDED

AS

(

# The seed:


FROM EMPLOYEES

WHERE ID NOT IN (SELECT MANAGER_ID FROM EMPLOYEES WHERE MANAGER_ID IS NOT NULL)

UNION ALL

# The recursive member:



GROUP BY M.ID, M.NAME, M.MANAGER_ID

)

# what we want to do with the complete result (the UNION):

SELECT * FROM EMPLOYEES_EXTENDED;

MySQL does not yet support WITH RECURSIVE, but it is possible to code a generic

stored procedure which can easily emulate it. Here is how you would call it:

CALL WITH_EMULATOR(

"EMPLOYEES_EXTENDED",

"


FROM EMPLOYEES


",




5 de 13 31/07/2015 4:01


",

"SELECT * FROM EMPLOYEES_EXTENDED",

0,

""

);

You can recognize, as arguments of the stored procedure, every member of the WITH

standard syntax: name of the temporary buffer, query for the seed, query for the

recursive member, and what to do with the complete result. The last two arguments - 0

and the empty string - are details which you can ignore for now.

Here is the result returned by this stored procedure:

+------+---------+------------+---------+

| ID | NAME | MANAGER_ID | REPORTS |

+------+---------+------------+---------+

| 72 | Pierre | 29 | 0 |

| 692 | Tarek | 333 | 0 |

| 4610 | Sarah | 29 | 0 |

| 29 | Pedro | 198 | 2 |

| 333 | Yasmina | NULL | 1 |

| 198 | John | 333 | 3 |

| 333 | Yasmina | NULL | 4 |

+------+---------+------------+---------+

7 rows in set

Notice how Pierre, Tarek and Sarah have zero reports, Pedro has two, which looks

correct... However, Yasmina appears in two rows! Odd? Yes and no. Our algorithm starts

from non-managers, the "leaves" of the tree (Yasmina being the root of the tree). Then

our algorithm looks at first level managers, the direct parents of leaves. Then at second

level managers. But Yasmina is both a first level manager (of the nonmanager Tarek) and




6 de 13 31/07/2015 4:01

contributes 1 direct/indirect report. The second tree branch contributes 4. The right

number, which we want, is the sum of the two: 5. Thus we just need to change the final

query, in the CALL:

CALL WITH_EMULATOR(

"EMPLOYEES_EXTENDED",

"


FROM EMPLOYEES


",

"




",

"

SELECT ID, NAME, MANAGER_ID, SUM(REPORTS)

FROM EMPLOYEES_EXTENDED

GROUP BY ID, NAME, MANAGER_ID

",

0,

""

);

And here is finally the proper result:

+------+---------+------------+--------------+

| ID | NAME | MANAGER_ID | SUM(REPORTS) |

+------+---------+------------+--------------+

| 29 | Pedro | 198 | 2 |




7 de 13 31/07/2015 4:01

| 692 | Tarek | 333 | 0 |

| 4610 | Sarah | 29 | 0 |

+------+---------+------------+--------------+

6 rows in set

Let's finish by showing the body of the stored procedure. You will notice that it does

heavy use of dynamic SQL, thanks to prepared statements. Its body does not depend on

the particular problem to solve, it's reusable as-is for other WITH RECURSIVE use

cases. I have added comments inside the body, so it should be self-explanatory. If it's

not, feel free to drop a comment on this post, and I will explain further.

# Usage: the standard syntax:

# WITH RECURSIVE recursive_table AS

# (initial_SELECT

# UNION ALL

# recursive_SELECT)

# final_SELECT;

# should be translated by you to

# CALL WITH_EMULATOR(recursive_table, initial_SELECT, recursive_SELECT,

# final_SELECT, 0, "").

# ALGORITHM:

# 1) we have an initial table T0 (actual name is an argument

# "recursive_table"), we fill it with result of initial_SELECT.

# 2) We have a union table U, initially empty.

# 3) Loop:

# add rows of T0 to U,

# run recursive_SELECT based on T0 and put result into table T1,

# if T1 is empty

# then leave loop,

# else swap T0 and T1 (renaming) and empty T1

# 4) Drop T0, T1




8 de 13 31/07/2015 4:01

# This is for *one* recursive table.

# It would be possible to write a SP creating multiple recursive tables.

delimiter |

CREATE PROCEDURE WITH_EMULATOR(

recursive_table varchar(100), # name of recursive table

initial_SELECT varchar(65530), # seed a.k.a. anchor

recursive_SELECT varchar(65530), # recursive member

final_SELECT varchar(65530), # final SELECT on UNION result

max_recursion int unsigned, # safety against infinite loop, use 0 for default

create_table_options varchar(65530) # you can add CREATE-TABLE-time options

# to your recursive_table, to speed up initial/recursive/final SELECTs; example:

# "(KEY(some_column)) ENGINE=MEMORY"

)

BEGIN

declare new_rows int unsigned;

declare show_progress int default 0; # set to 1 to trace/debug execution

declare recursive_table_next varchar(120);

declare recursive_table_union varchar(120);

declare recursive_table_tmp varchar(120);

set recursive_table_next = concat(recursive_table, "_next");

set recursive_table_union = concat(recursive_table, "_union");

set recursive_table_tmp = concat(recursive_table, "_tmp");

# If you need to reference recursive_table more than

# once in recursive_SELECT, remove the TEMPORARY word.

SET @str = # create and fill T0

CONCAT("CREATE TEMPORARY TABLE ", recursive_table, " ",

create_table_options, " AS ", initial_SELECT);

PREPARE stmt FROM @str;

EXECUTE stmt;




9 de 13 31/07/2015 4:01

EXECUTE stmt;

SET @str = # create T1

CONCAT("CREATE TEMPORARY TABLE ", recursive_table_next, " LIKE ", recursive_table);


EXECUTE stmt;

if max_recursion = 0 then

set max_recursion = 100; # a default to protect the innocent

end if;

recursion: repeat

# add T0 to U (this is always UNION ALL)

SET @str =

CONCAT("INSERT INTO ", recursive_table_union, " SELECT * FROM ", recursive_table);


EXECUTE stmt;

# we are done if max depth reached

set max_recursion = max_recursion - 1;

if not max_recursion then

if show_progress then

select concat("max recursion exceeded");

end if;

leave recursion;

end if;

# fill T1 by applying the recursive SELECT on T0

SET @str =

CONCAT("INSERT INTO ", recursive_table_next, " ", recursive_SELECT);


EXECUTE stmt;

# we are done if no rows in T1

select row_count() into new_rows;

if show_progress then

select concat(new_rows, " new rows found");

end if;




10 de 13 31/07/2015 4:01

# Prepare next iteration:

# T1 becomes T0, to be the source of next run of recursive_SELECT,

# T0 is recycled to be T1.

SET @str =

CONCAT("ALTER TABLE ", recursive_table, " RENAME ", recursive_table_tmp);


EXECUTE stmt;

# we use ALTER TABLE RENAME because RENAME TABLE does not support temp tables

SET @str =

CONCAT("ALTER TABLE ", recursive_table_next, " RENAME ", recursive_table);


EXECUTE stmt;

SET @str =

CONCAT("ALTER TABLE ", recursive_table_tmp, " RENAME ", recursive_table_next);


EXECUTE stmt;

# empty T1

SET @str =

CONCAT("TRUNCATE TABLE ", recursive_table_next);


EXECUTE stmt;

until 0 end repeat;

# eliminate T0 and T1

SET @str =

CONCAT("DROP TEMPORARY TABLE ", recursive_table_next, ", ", recursive_table);


EXECUTE stmt;

# Final (output) SELECT uses recursive_table name

SET @str =

CONCAT("ALTER TABLE ", recursive_table_union, " RENAME ", recursive_table);


EXECUTE stmt;




11 de 13 31/07/2015 4:01

Publié par Guilhem Bichot à 06:17

Libellés : optimizer, stored procedure, subquery, with_recursive

EXECUTE stmt;

# No temporary tables may survive:

SET @str =

CONCAT("DROP TEMPORARY TABLE ", recursive_table);


EXECUTE stmt;

# We are done :-)

END|

delimiter ;

In the SQL Standard, WITH RECURSIVE allows some nice additional tweaks

(depth-first or breadth-first ordering, cycle detection). In future posts I will show how to

emulate them too.

2 commentaires:

swanhart 20 novembre 2013 03:20

Very cool.

Répondre

Renato Kolbe 2 juin 2014 11:29

Great ! Thanks !!!!! It might be very helpfull !

Répondre




12 de 13 31/07/2015 4:01

Article plus récent Article plus ancienAccueil

Inscription à : Publier les commentaires (Atom)

Commentaire :

Publier

The views expressed on this blog are my own and do not necessarily reflect the views of Oracle. Modèle Watermark. Fourni par Blogger.




13 de 13 31/07/2015 4:01

Documents

About MySQL Development (Optimizer Mainly)- WITH RECURSIVE and MySQL