Uri Margalit

Why Retail Companies Can’t Afford Database Downtime

2

• You will be on mute for the duration of the event

• We are now talking so please type a message in the Questions box in the Control Panel if you can’t hear us (please check your speakers and GoToWebinar audio settings first)

• If you have questions during the session, please submit them on the Q&A bar on your gotowebinar dashboard

• A recording of the full webinar will be put up online

Before We Begin

3

Presenter

Uri Margalit

Director of Products at DBmaestro

About DBmaestro

• Founded in 2008, product launched in 2010• Founded by Yariv Tabac and Yaniv Yehuda• Headquartered in Israel, Global Operations

5

• Retail challenges• The database uniqueness• Real life tales• Common approaches analysis• The solution from DBmaestro TeamWork

Agenda

6

Retailers are smack in the middle of three to five years of major changes that will redefine the industry.

I would say the last time there was this big [of] a change in the industry was in 1960, when many of the big boxes opened up and really opened the huge discount department store. The biggest change before that - the invention of modern retailing in postwar America.Alison Paul, vice chairman and U.S. retail and distribution leader at consultancy Deloitte

Retail’s Unique Challenges

“

7

• Business• Downtime = Loss revenue• Heavily regulated – large amount of customer data

stored across multiple locations• Migration from physical stores to web-based online

stores using Augmented Reality experiences• Freeze toward year-end changes for fear of

download and delaying agility because of lack of confidence in process

• Rapidly change & Agile market needs in a highly seasonal environment – downtime is critical

• Streamline development & operations team

Retail’s Unique Challenges

8

• Technical• Complex architecture and internal dependencies• Responding quickly to business changes• 3rd party integrations• Same metadata across many environments• Heterogenic environments• Changing applications structures from packaged

applications to manage physical spaces and stock to more customized/home grown web-facing applications

• Streamline geographically diverse development & operational teams

Online Gaming Unique Challenges

9

Why Manage Change?

Copyright@2008, Juniper Networks, Inc.

50% 40% of changes fail

of unplanned downtime is due to Change

80%

of this, half is due human errors50%

More than

10

• DECM - Database Enforced Change Management• Part of ALM (Application Lifecycle Management)

solutions

DECM & ALM

11

• The database holds your essential information

• Changes can impact the entire system• Need to be synchronized with other changes• Often overlooked

Database is a Key Component

12

• There is more to a database than SQL scripts• Schema structure• Code• Content and meta-content• Internal dependencies• …

• Ensure that changes are not made without authorization

• Ensure no out-of-process changes

Reaching Inside the Database

13

• Old adage but true• The database is often “forgotten” and

therefore can become the weakest link• Essential from a compliance point of view• Should be the strongest link

The Weakest Link In a Chain

“

“

It was difficult to track who made a change to a database object and what change they made.(working-around and failing to enforce usage of file based version control)

Sr. DBA @ Large USA Bank

It took hours to get releases working. some changes were not documented and left out. we actually preferred crashes in integration. It is much worse when something works, but works wrong. in production…(manual and error prone releases. Change control & change scope issues)

Sr. R&D Manager @ Credit Card company

The Database is a constraint

We recently had a disaster - the script in the version control was not updated and when executed in production, ran the wrong revision. That cost tens of thousands of $(an out-of-process update to QA that was not properly tracked. Same for integ/stage/prod…)

DBA @ Algo Trading company

We had multiple releases to production every day. That is one release a week with multiple follow up fixes, and yet more fixes(code overrides, partial versions, wrong versions – all pushed to production)

CTO @ Credit Card company

“

“

The Database is a constraint

We had an incident where a trigger was not correctly implemented during a code release. a trigger from a previous build was used instead which was only detected on Tuesday morning on the first business day after our code release. this was a customer-facing application and made our team look, and feel, bad about the release.

we realized that we needed to bring more discipline and rigor to our database changes.(manual process are hard to repeat over and over without errors)

Sr. DBA @ Large USA Bank

“The Database is a constraint

17

Common DB Development Methods

• Treating database scripts as native code• Small scripts• One large script

• Automatically generated scripts (Compare & Sync)

18

Two isolated Processes

Version Control Process (file based)

Development Process

(real database)

Check-Out Script

Modify Script

Get updated Script from DB

Check-In Script

Compile Scriptin DB

Debug Scriptin DB

?

??

?

A

A’

Version control repository is out of sync from the tested database and cannot acts as Single Source of Truth

19

Scripts & Version Control

• Challenges… • Working on the wrong revisions • Code-overrides• Scripts do not always find their way to the version control

solution• Out of process updates go unnoticed• Hard to locate outdated update scripts

• Playing safe? what we really need (manual steps):• The actual code of the object• The upgrade script• A roll-back script

20

Scripts are static…

• Scripts, unless super sophisticated:• Unaware of changes made in the target environment• Time passed from their coding to the time they are run• Potentially overriding production hot-fixes or work done

in parallel by another team

• Content changes are very hard to manage• Metadata & lookup content does not practically fit into

the VC• In most cases they are simply not managed

 

21

Testing Scripts

• Single object based scripts• Hard to test in their entirely (holistically)• Hard to test due to colliding dependencies• Need to run in a specific order…

• Large multi object based script• Represents the entire update - can deal with

dependencies• Much harder to deal with project scope changes• Hard to mange – a big list of commands

22

X1.11.1.11.11.21.31.41.51.61.7

Build Once Deploy Many (Scripts)

Int QA Pre Prod Prod

Database Deploy Script

Environment

Re-Base (due to defects)

DevDev

DevModel

1.1 1.2

1.2 1.3

1.3 1.4

1.4 1.5

1.5 1.6

1.6 1.7

1.11.11.41.7

1.1 1.2

1.2 1.3

1.3 1.4

1.4 1.5

1.5 1.6

1.6 1.7

1.1 1.2

1.2 1.3

1.3 1.4

1.4 1.5

1.5 1.6

1.6 1.7

Out of Process Change

X

X

X

X

X

? 1.1.1

X

23

Scripts – Bottom Line

• Overhead time• No trust in the version control repository• Difficult and very complex to maintain• No agility to business changes

24

Compare & Sync

• Challenges… • Handling out of process changes• Handling parallel development• Handling conflicts• Not an integral part in development phase - version

control repository is NOT the single source of truth

Source vs. Target

Action

= No Action

≠ ? We do not have all of the information

27

1.11.21.31.41.51.61.7

Build & Deploy On Demand(Compare & Sync)

*

Int QA Pre Prod Prod

Database Deploy Script

Environment* Execute the same script being executed at the Stage environment

Re-Base (due to defects)

DevDev

DevModel

1.1 1.2

1.2 1.3

1.3 1.4

1.4 1.5

1.5 1.6

1.6 1.7

1.1 1.4

1.4 1.7

1.1.1 1.7

1.1 1.1 1.11.41.7

File Based Version Control

Out of Process Change

1.1.11.7 1.1.11.7

28

Deployment Automation Safety Net

An index exists in Target (Production) but not in Source (QA). What should we do? Drop the index or not?

Baseline Aware AnalysisSource vs. Target

Action

= No Action

≠ ?

Source vs. Baseline

Target vs. Baseline

Action

= = No Action

≠ = Deploy Changes

= ≠ Protect Target

≠ ≠ Merge Changes

You do not have all of the information

With Baselines and 3 way analysis the unknown is now known

Simple Comp & Sync

29

So why not move forward?

30

• Database Enforced Change Management• Database version control• Plugs into the ALM (change request, tickets & work items)• Database change impact analysis• Database deployment automation

• DevOps Solution for databases • Deployment, rollback & recovery• Plugs into release management

What is DBmaestro TeamWork

31

With DECM - One Enforced Process

32

• Raise red flags on conflicts • Support out-of-process changes• Utilize baseline aware analysis• Understand the nature of the changes

Safety Net Deployment Automation

Source vs. Baseline

Target vs. Baseline

Action

= = No Action

≠ = Deploy Changes

= ≠ Protect Target

≠ ≠ Merge Changes

33

• Development• Database change repository• Follow SCM best practices (Check-Out/Check-In) • Documentation of who did what, where, when & why

• Operation• Integrated deployment engine• Business level audit• Roles & responsibilities enforcement

• Management• Complete visibility into changes in progress• Management reports• No silos

Benefits – By Role

34

• Productivity improvements• No manual work and less reworking• No time spent in testing deployment scripts• Supports parallel development

• Quality improvements• No code overrides• No promoting wrong code• Supports partial releases

• Less deployment failures• Accurate & complete deploy script• Supports task based development

• Auditing & Regulations• A complete documentation of who did what,

where, when & why

Other Benefits

35

Focusing on changes rather than managing changes and dealing with re-work, boosted overall productivity of 200-300 developers. We estimate we were able to do 15% more with the same resources.

We went from several fix-centric deployments a day, to one feature-centric deployment a week.

The amount of incidents in production has declined as well. We had 20% less incidents. CIO @ Credit Card company

Testimonials - Efficiency

“

36

Regulation requirement - SOX & derived 357

Auditing - change management approach with change management auditing. You have to do it anyway, do it automatically and efficiently.

Change management in production itself is regulatory required (ITIL). But you cannot ensure it without managing the whole process starting at Dev.

We have to comply to regulation - but the business benefits from it. CIO @ Credit Card company

“Testimonials - Regulation

37

Selected Customers

And more…

38

Customer Testimonials

“Massachusetts Department of Education uses DBmaestro TeamWork to reduce database change deployment time by up to 95%”Mr. Dan Boyle, Supervisor of Database Administration & Development

“Isracard has increased the throughput of our development team immensely and along with that our team is more motivated and satisfied on the job. With the TeamWork™ proprietary Automation Module we are able to deploy changes between environments automatically and this reduces the number of human errors and increases the production system's stability”Ofer Krichman, Head of Infrastructure Software Department , Isracard LTD

“DBmaestro TeamWork facilitated the database development process in the Ministry of Education in the current IT development projects. The ROI of DBmaestro TeamWork is doubtless...”Mrs. Sofia Mintz, CIO

“By implementing the DBmaestro solution, We can finally take control of our databases. By Integrating the application alongside our source control and continuous integration paradigm, We are able to deploy our application within minutes.”Mr. Jonathan Bar-Sela, Development Manager, Clalbit systems

“DBmaestro TeamWork, provides us full control on our databases environments. Having our schemas sync between all environments made a big change in the DBA’s and developers efficiency. TeamWork integration to MS-TFS, provides us End-to-End products development life cycle”Mr. Shahaf Azriely, Director of IT, IncrediMail Ltd.

“By implementing TeamWork™, Sura finally has the ability to apply the SCM best practices on database development…Finally, Sura's database deployment will be quick and efficient with an ideal rollback option.”Juan David Perez Chief Architect, Sura Insurance

39

Summary

• Enforce best practices for database development• Closely manage your database development• Know who can do what, where, when & why• Build a safety net to catch problems early• Reduce deployment issues• Enable automated deployments

40

Demo

DeploymentScreen shots

41

Integrations

• ALM Solutions• TFS• RTC

• Task Management Systems• Clear Quest• Perforce• JiRA• TeamForge• Any tool, using open API

• Continuous Integration• Jenkins• Bamboo• Any tool, using web-services

• Release Automation• IBM UrbanCode Deploy• CA Release Automation• Serena• BMC DevOps• Any tool, using web-services

• IDEs• Oracle SQL Developer• SQL Server Management Studio• Visual Studio• JDeveloper

• Other Database Tools• Delphix (Virtual Branches)• DM Suite (Content Masking)• Oracle SQL Developer (Unit Test

& Code Coverage)

42

Merge Dev1 to ForkMerge to dev2

Dev2

Dev1Merge to dev1

Merge Dev2 to Fork

Trunk

Merge Dev1 to Fork

Merge Dev2 to Fork

DBVC

Fork

Fork

Fork

Fork

Merge, Rebase, Fork, …

43

Use of Baseline

Int / QA

Dev1

Label/Tag/BaselineBackup Label/TagMerge

1

Ver 1.0

Ver 1.0

2 3

MS 1.1Merged - B

Before MS 1.1

MS 1.1Merged - M

Branch Merging MS 1.1

MS 1.1

7 8

MS 1.2 Merged - B

Before MS 1.2

Branch Merging MS 1.2 MS 1.2

Merged - MMS 1.2

Dev2 Ver 2.0 MS 2.1Merged - B

Before MS 2.1 MS 2.2 Merged - B

Before MS 2.2

4 5 6 9 10

MS 2.1Merged - MBranch

Merging MS 2.1

Branch Merging MS 2.2

MS 2.2 Merged - M

MS 2.1 MS 2.2

Deploy Source Target Baseline

1 Int / QA (Live) Dev1 …

2 Int / QA (MS 1.1) Dev1 (Live) Dev1 (Ver 1.0)

3 Dev1 (MS 1.1 Merged - B) Int / QA (Line) Int / QA (MS 1.1)

…