Upload
oleksandr-zarichnyi
View
47
Download
0
Embed Size (px)
Citation preview
Minimizing Magento Upgrade
Downtime
Legal Disclaimer
Copyright © 2015 Magento, Inc. All Rights Reserved.
Magento®, eBay Enterprise™ and their respective logos are trademarks, service marks, registered trademarks, or registered service marks of eBay, Inc. or its subsidiaries. Other trademarks or service marks contained in this presentation are the property of the respective companies with which they are associated.
This presentation is for informational and discussion purposes only and should not be construed as a commitment of Magento, Inc. or eBay Enterprise (“eBay Enterprise”) or of any of their subsidiaries or affiliates. While we attempt to ensure the accuracy, completeness and adequacy of this presentation, neither Magento, Inc., eBay Enterprise nor any of their subsidiaries or affiliates are responsible for any errors or will be liable for the use of, or reliance upon, this presentation or any of the information contained in it. Unauthorized use, disclosure or dissemination of this information is expressly prohibited.
Oleksandr Zarichnyi
Developer, Magento ECG
Magento ECG
Magento's Expert Consulting Group (ECG) helps Magento Enterprise Edition merchants and Solution Partners maximize their success.
magento-ecgmagento.com/
consulting
Minimizing Magento Upgrade
Downtime
400%
1.8 1.9 1.10 1.111.12
1.131.14
DB Upgrade
Schema changes:• New columns, keys• Modify column, storage engine
Data changes:• Update records• Insert new records• Migrate data
– Promotion rules– Customers– URL keys
Time
DB size,# of schema
changes
DB Upgrade Time Dependencies
We were here
Approaches
2. Upgrade Replay 3. Optimized Upgrade
• run Magento upgrade scripts
• single process 5 hrs
1.5 hrs
• run SQL queries fromthe log file
• multiple processes
1. Standard Upgrade
1.8 1.14
59 hrs
Downtime
Magento Upgrade Scope
• Configure new environment
• Prepare the codebase
• Go live
• Dry run
• Fix code issues
Dry Run
• Enable Magento SQL queries log
//in lib/Varien/Db/Adapter/Pdo/Mysql.php
protected $_logAllQueries = true;protected $_debug = true;
• Run Magento standard upgrade
$ screen -S upgrade$ n98-magerun.phar sys:setup:run
Output: var/log/pdo_mysql.log
Standard Upgrade: Logging Queries
59 hours
Standard Upgrade
Standard DB Upgrade Bottlenecks
• Unnecessary queries• Duplicated queries• Non-optimized queries• Multiple slow ALTER TABLE• Single process execution
Analyzing pdo_mysql.log
TRUNCATE
CREATE
DROP
DELETE
SET
INSERT
UPDATE
SELECT
ALTER
DESCRIBE
SHOW
0 5000 10000 15000 20000 25000 30000
A Really, Really, Really, Really Slow Query
`salesrule_product_attribute` in EE 1.12INSERT INTO `salesrule_product_attribute` explain SELECT sr.rule_id, cw.website_id, cg.customer_group_id, ea.attribute_id FROM `salesrule` AS sr INNER JOIN `core_website` AS cw ON FIND_IN_SET(cw.website_id, sr.website_ids) INNER JOIN `customer_group` AS cg ON FIND_IN_SET(cg.customer_group_id , sr.customer_group_ids) INNER JOIN `eav_attribute` AS ea ON ea.entity_type_id = 4 WHERE sr.conditions_serialized LIKE CONCAT('%s:32:\"salesrule/rule_condition_product\";s:9:\"attribute\";s:', LENGTH(ea.attribute_code), ':\"', ea.attribute_code, '\"%') OR sr.actions_serialized LIKE CONCAT('%s:32:\"salesrule/rule_condition_product\";s:9:\"attribute\";s:', LENGTH(ea.attribute_code), ':\"', ea.attribute_code, '\"%')");
20
hours
CREATE TEMPORARY TABLE tt1AS SELECT sr.rule_id, ea.attribute_id FROM `salesrule` AS sr INNER JOIN `eav_attribute` AS ea ON ea.entity_type_id = 4 WHERE CONCAT(sr.conditions_serialized, sr.actions_serialized) LIKE CONCAT('%s:32:\"salesrule/rule_condition_product\";s:9:\"attribute\";s:', LENGTH(ea.attribute_code), ':\"', ea.attribute_code, '\"%') AND ea.attribute_code IN ('sku', ’coupon_exclude');
INSERT INTO `salesrule_product_attribute` SELECT sr.rule_id, cw.website_id, cg.customer_group_id, tmp.attribute_id FROM `salesrule` AS sr INNER JOIN tt1 AS tmp ON tmp.rule_id = sr.rule_id INNER JOIN `core_website` AS cw ON FIND_IN_SET(cw.website_id, sr.website_ids) INNER JOIN `customer_customer_group` AS cg ON FIND_IN_SET(cg.customer_group_id, sr.customer_group_ids);
A Really, Really, Really, Really Slow Query Fixed
SKU and coupon_exclud
e only
30
seconds
• Removes redundant and duplicate queries.• Combines similar ALTER TABLE queries.• Runs queries in batches in multiple processes
using PHP POSIX capabilities.
Approach 2: Upgrade Replay Script
magento-ecg/magento-upgrade-replay
5 hoursUpgrade Replay
Optimized Upgrade: Dealing with Dependencies
• Drop all foreign keys• Run schema update queries grouped by table• Add foreign keys
1.8 1.9 1.10 1.111.12
1.131.14
Optimized Upgrade: Skipping Intermediate Steps
Optimized Upgrade: Creating a Sync Script
• Install Vanilla Magento 1.14• Compare 1.14 DB with the current 1.8 DB• Create a sync script
Step 1. Specify Vanilla 1.14 DB as a source
Step 2. Specify Current 1.8 DB as a target
Step 3. Run comparison
Step 4. Save synchronization script to file
Optimized Upgrade: Running Queries in Parallel
zlik/magento-resque
$ QUEUE='upgrade' COUNT=24 ./bin/resque
• Run multiple workers
$ php resque.php sql/queries.sql
• Add queries to the queue
PHP-Redis Job Queue
Optimized Upgrade: Data Clean Up
Data to be cleaned up:• Old sales quotes• Visitor logs• Recently viewed/compared products• Expired event-specific categories• Expired promotion rules• …
Optimized Upgrade: Fast Delete Data Strategies
• TRUNCATE to delete all rows
• DELETE with limit
• Move –> Rename
CREATE TABLE `sales_flat_quote__upg` LIKE `sales_flat_quote`; INSERT INTO `sales_flat_quote__upg` SELECT * FROM `sales_flat_quote` WHERE `updated_at` >= '2015-01-01'; RENAME TABLE `sales_flat_quote` TO `sales_flat_quote__old`, `sales_flat_quote__upg` TO `sales_flat_quote`;DROP TABLE `sales_flat_quote__old`;
Optimized Upgrade: Clean Up in Parallel
<?php
exec('php –f clean_up_logs.php > clean_up.log 2>&1 &');
exec('php –f clean_up_quotes.php > clean_up.log 2>&1 &');
exec('php –f clean_up_visitors.php > clean_up.log 2>&1 &');
1,5 hours
Optimized Upgrade
Can We Do Better?
“Zero-downtime” approach:• Enable the change log for every table
in the source database;• Run the optimized upgrade of the target
database;• Put the current production down;• Migrate changes to the target database;• Start the new production.
Lesson Learned
More time spent preparing – less downtime
Thank [email protected]
github.com/zlikgithub.com/magento-ecg