The challenges and pitfalls of database deployment automation

  • View

  • Download

Embed Size (px)



Text of The challenges and pitfalls of database deployment automation

  • 1. Yaniv Yehuda The Challenges and Pitfalls of Database Deployment Automation

2. 2 You will be on mute for the duration of the event We are now talking so please type a message in the Questions box in the Control Panel if you cant hear us (please check your speakers and GoToWebinar audio settings first) There will be a Q+A session at the end, you can start submitting you questions on the Q&A bar on your gotowebinar dashboard. A recording of the full webinar will be put up online Before We Begin 3. 3 Presenter Yaniv Yehuda CTO, Co-Founder at DBmaestro 4. About DBmaestro Founded in 2008, product launched in 2010 Founded by Yariv Tabac and Yaniv Yehuda Headquartered in Israel, Global Operations 5. 5 Doing better with less Reacting quickly to market needs Getting ahead of competition Just cant wait 6 months for that next release Agile Development Process Automation DevOps Agile world 6. Dealing with Risk Smaller and more focused changes are easier to manage (Agile) Automation of repeating tasks lowers risk of (human) error Development and Operations should work in synergy (DevOps) 7. 7 Continuous Integration Continuous Delivery Continuous Deployment Automation is the Name of the Game 8. 8 Principles and practices Focus on streamlining development Developers integrate code into shared repository Each check-in is verified Automated builds Automated tests High visibility Easier & quicker to prevent and find problems, less back tracks => short integrations What is Continuous Integration? 9. 9 Next step after continuous integration Becoming lean, and even more Agile Make sure each change is releasable Develop-> build-> test-> move to staging-> acceptance test Build a process to release with a push of a button Deploy to production-> test production Actual deployment to production is manually actuated => Ensure risk mitigation and high efficiency And Continuous Delivery? 10. 10 Rapid changes Reacting quickly to market needs Getting ahead of competition Fewer changes backed out Better collaboration Fewer defects Ultimately better service Happy customers Profitability Why Continuous Delivery? 11. 11 Team and process Version everything Automate your tests Fix it, properly (no out of process changes!) Automate your deployments Create the deployment pipeline Focus points 12. 12 But 13. 13 The database holds your essential information Any changes can impact the entire system Need to be synchronized with other changes Often overlooked Database is a Key Component 14. 14 There is more to a database than SQL scripts Schema structure Code Content and meta-content Internal dependencies Ensure that changes are not made without authorization Ensure no out-of-process changes Reaching Inside the Database 15. 15 Old adage but true The database is often neglected and therefore can become the weakest link Essential from a compliance and business point of view Should be the strongest link The Weakest Link In a Chain 16. 16 Silos exist Dont always communicate effectively Need to share knowledge Need to follow same procedures & best practices Developers and DBAs 17. 17 Real-life tales 18. 18 Real-life tales it was difficult to track who made a change to a database object and what change they made. (working-around file based version control) it took hours to get releases working. some changes were not documented and left out. we actually preferred crashes in integration. It is much worse when something works, but works wrong. in production (manual and error prone releases) 19. 19 Real-life tales We recently had a disaster - the script in the version control was not updated and when executed in production, ran the wrong revision. That cost tens of thousands of $ (an out-of-process update to QA that was not properly tracked) We had multiple releases to production every day. That is one release a week with multiple follow up fixes, and yet more fixes (code overrides, partial versions, wrong versions all pushed to production) 20. 20 Real-life tales we had an incident where a trigger was not correctly implemented during a code release. a trigger from a previous build was used instead which was only detected on Tuesday morning on the first business day after our code release. this was a customer-facing application and made our team look, and feel, bad about the release. we realized that we needed to bring more discipline and rigor to our database changes. (manual process are hard to repeat over and over without errors) 21. 21 Challenges Un-coordinated Process Silos in development Deployment delays Out-of-Process Changes Errors in production Lack of confidence in automation Code overrides Different methodologies Lack of history Missing changes wrong versions 22. 22 Root Causes: Manual script based version control process Deployment tools unaware of version control No change process red-flags 23. 23 Two isolated Processes Version Control Process (file based) Development Process Check-Out Script Modify Script Get updated Script from DB Check-In Script Compile Script in DB Debug Script in DB ? ? ? ? A A 24. 24 Scripts Pit falls 25. 25 Scripts & Version Control Challenges Code-overrides Working on the wrong revisions Scripts do not always find their way to the version control solution Out of process updates go unnoticed Hard to locate outdated update scripts Playing safe? what we really need: The actual code of the object The upgrade script A roll-back script 26. 26 Testing Scripts Single object based scripts Hard to test in their entirely (holistically) Hard to test due to colliding dependencies Need to run in a specific order Large multi object based script Represents the entire update - can deal with dependencies Much harder to deal with project scope changes Hard to mange a big list of commands 27. 27 X Scripts Build Once Deploy Many Int QA Stage Prod Database Deploy Script Environment Re-Base (due to defects) Dev Dev Dev Model 1.1 1.2 1.2 1.3 1.3 1.4 1.4 1.5 1.5 1.6 1.6 1.7 1.1 1.2 1.2 1.3 1.3 1.4 1.4 1.5 1.5 1.6 1.6 1.7 1.1 1.2 1.2 1.3 1.3 1.4 1.4 1.5 1.5 1.6 1.6 1.7 Out of Process Change X X X X X ? 1.1.1 X 28. 28 Scripts are static Scripts, unless super sophisticated: Unaware of changes made in the target environment Time passed from their coding to the time they are run Potentially overriding production hot-fixes or work done in parallel by another team Content changes are very hard to manage Metadata & lookup content does not practically fit into the VC In most cases they are simply not managed 29. 29 Dealing with challenges Coordinated ProcessTraceability Start in the Beginning No Out-of-Process Changes Impact Analysis Automation Task Based Development Well Defined Processes 30. 30 Version Control - One Enforced Process 31. 31 Dealing with challenges Integrated Version Control process Leverage proven version control best practices Forcing check in & out for changes Labels etc.. No code-overrides Always working with the correct revision All changes are documented Always know who did what, when, why and from where No out-of-process changes Supporting structure, code and content No time spent on manual coding of the change scripts 32. 32 Bonus Points Task based development Correlate each database change with a change request Task ID Work Item Trouble Ticket CR etc enables task based deployments Partial deployments (a feature, a collection of bugs, etc) Scope changes easily synced between code and database 33. 33 Change Policy Enforcement 34. 34 Change Policy Enforcement 35. For deployment automation we need: Automatic, repeatable & safe Using tools make sense 36. 36 Build & Deploy On Demand * Int QA Stage Prod Database Deploy Script Environment* Execute the same script being executed at the Stage environment Re-Base (due to defects) Dev Dev Dev Model 1.1 1.2 1.2 1.3 1.3 1.4 1.4 1.5 1.5 1.6 1.6 1.7 1.1 1.4 1.4 1.7 1.1.1 1.7 1.1 1.1 File Based Version Control Out of Process Change 37. Real World Deployment Automation Test cases using compare & sync tools: An index exists in source (QA) but not in target (Production) What should we do? Add the index or not? 38. 38 Compare & Sync tools Safe to automate? Sure 39. Deployment Automation An index exists in Target (Production) but not in source (QA) What should we do? Drop the index or not? 40. 40 Compare & Sync tools Safe to automate? No. Requires manual inspection 41. 41 Safe? Simple, right? NO! we are going to BREAK production without even knowing 42. 42 Why break production??? A compare & sync tool: Is unaware of any changes that occurred before the time it ran Has no knowledge of changes that took place at the target environment Does not leverage version control for more information Unable to deal with conflicts & merges between different teams Requires manual inspection Requires detailed knowledge regarding each change as part of the process So no automation as automating problems into production is a major risk!!! 43. 43 We need to leverage version control into deployment decisions 44. Safety Net For Deployment Automation Source vs. Target Action = No Action ? Source vs. Baseline Target vs. Baseline Action = = No Action = Deploy Changes = Protect Target Merge Changes You do not have all of the information With Baselines and 3 way analysis the unknown is now known Simple Compare & Sync Baseline aware Analysis 45. 45 Protecting Target Environment Development Baseline Previous Label / Production Golden Copy Production No index in baseline => we should protect the NEW index on production!!! (Protect Target) 46. 46 Protecting Target Environment Development