Embed Size (px)
DESCRIPTION
You never develop code without version control, why do you develop your database without it? With Liquibase, database changes are stored in human XML-files and committed to the source control system. Changes are applied to the developers local databases. As changes are committed they are distributed to all other environments including all developers local databases, test databases, staging databases, and even to production databases. This presentation will introduce you to Liquibase and the topic database change management. We will also present some advanced topics based on real life experience and a few tips and tricks as well Rikard Thulin, Squeed and Roger Nilsson, Altran
Citation preview
Database Change Management
Roger Nilsson, AltranRikard Thulin, Squeed
Rikard ThulinHas been working as a software engineer for 17 years, mostly as an consultant with focus on the Java platform
e-mail: [email protected], twitter: @rikard_thulin, blog: http://squeed.com/blog
Roger NilssonHas been working as a software engineer and consultant for 15 years, with focus on the Java platform
e-mail: [email protected]
About the Presenters
Questions?
Questions during the presentation is encouraged
There will be a Q&A if time permits at the end of the presentation
The Question?
You never develop code without version
control,
why would you develop your database without it?
Liquibase at a glance
Liquibase is an Apache 2.0 Licensed tool for tracking, managing and applying database changes
Database changes are stored in an XML (or JSON, YAML, plain-ish SQL) file that is version controlled and database independent
Liquibase managed and applies changes to the database
Major Concepts - Changeset
A changeset is uniquely identified change that should be applied to the database
When Liquibase runs, it queries the DATABASECHANGELOG table for the changesets that are marked as executed and then executes all changesets that have not yet been executed
Major Concepts - Changeset
Major Concepts - Changelog file
The changelog file contains Change Sets or references to other Changelog files
Changelog files can be be arbitrarily nested for better management
Major Concepts - Changelog file
<databaseChangeLog> <include file="changelogs/jira-937.xml"/> <include file="changelogs/jira-1023.xml"/> <include file="changelogs/jira-1104.xml"/>
<!-- Stored Procedures --> <include file="procedures/uspUpdateSearchIndex.sql"/>
</databaseChangeLog>
Major Concepts - Changes
Each changeset generally contains a change which describes the change/refactoring to apply to the database
Liquibase supports both descriptive changes that generate SQL for supported databases and raw SQL
Major Concepts - Changeset
Major Concepts - Preconditions
Preconditions can be applied to either the changelog as a whole or individual change sets
columnExists, tableExists, viewExists, foreignKeyConstraintExists, sqlCheck, and more...
If a precondition fails, Liquibase will stop execution
Major Concepts - Contexts
Contexts can be applied to changesets to control which are ran in different environments
For example, some changesets can be tagged as "production" and others as "test"
Use Contexts only when there is a good reason
● Update rollbacks● Database ”diff“s● Generating starting change logs from
existing databases● Generating database change documentation● Code branches and merging
Liquibase feature set
Database Change Management
Database deltas are part of the code change
The correlation between code revisions are ensured
Allows the automatisation of database changes
Keeps all environments consistent
Allows reverting a previous version of a system
Why Liquibase?
Source Code
Why Liquibase?
8 developers with 8 local databases/schemas
Development server
Test / QA server
CI Server
Staging Server Node 1
Production Node 1
Staging Server Node 2
Production Node 2
Why Liquibase?
8 developers with 8 local databases/schemas
Development server
Test / QA server
CI Server
Staging Server Node 1
Production Node 1
Staging Server Node 2
Production Node 2
The four steps
1. Write your database change set2. Run Liquibase locally to test the SQL3. Commit your source code and change set4. Deploy application and database changes
Liquibase plays well when merging and working with branches
Structural Refactorings● Add Column● Rename Column● Modify Column● Drop Column● Alter Sequence● Create Table● Rename Table● Drop Table● Create View● Rename View● Drop View● Merge Columns● Create Stored Procedure
Data Quality Refactorings● Add Lookup Table● Add Not-Null Constraint● Remove Not-Null Constraint● Add Unique Constraint● Drop Unique Constraint● Create Sequence● Drop Sequence● Add Auto-Increment● Add Default Value● Drop Default Value
Referential Integrity Refactorings● Add Foreign Key Constraint● Drop Foreign Key Constraint● Drop All Foreign Key Constraints● Add Primary Key Constraint● Drop Primary Key Constraint
Non-Refactoring Transformations● Insert Data● Load Data● Load Update Data● Update Data● Delete Data● Tag Database● Stop
Architectural Refactorings● Create Index● Drop Index
Custom Refactorings● Modifying Generated SQL● Custom SQL● Custom SQL File● Custom Refactoring Class● Execute Shell Command
Supported Refactorings
Running Liquibase
● Manual○ Command Line○ Ant○ Maven
● Automatic ○ Spring○ Servlet Listener○ CDI Environment
Tips and Tricks - real life experience
You must have a manual way to run Liquibase (command line, ant or maven) during development of changesets
Bundle your database changes into your deployment artifact (spring, servlet listener or CDI)
It is (mentally) scary to do, but it works!
Demo
1. Update Database with pending changesets
2. Rollback previous changes
Tips and tricks - structure
Think about the structure of your changelogs
Based on experience, when the changelog xml file is 10.000 lines long it is gets ugly. It is much more difficult when merging and branching.
Have one master changelog that includes smaller changelogs in turn, divided by release or issue
changelog.xml changelog.xmljira-1.xml changelog-1_0.xmljira-4.xml changelog-1_1.xml
Tips and tricks - structure
Liquibase should foremost be used for structural changes but it is also possible to use it for configuration data and for test data
Keep in mind that the use case for these are very different!● Structural changes should be applied together with the
corresponding code● Configuration data has en dependency to the database
structure, but is not applied at any time● Test data has a dependency to the database structure,
may have preconditions and is typically applied by a CI server nightly
Tips and tricks - structure
● changelog.xml for structural changes, applied when the system is deployed
● test-data.xml for test data, applied nightly be CI server
● configuration-data.xml for system configuration changes, applied manually when needed
When executing update or rollback there is a handy option which do not apply changes against the database but instead save the resulting SQL
This can be usable when*● the resulting SQL needs to be tweeked● having the SQL approved by a DBA● SOX compliance.
*) Not a recommended best practices by the authors
Advanced TopicsSQL Output
Generating RFC to DBA automatically using Jenkins
REQUEST FOR CHANGE for sqlserver://sqlprod
Planning: 1. Changeset esc-1729.xml::1::rikard::(Checksum: 3:be601b760eaccc5d864da5b3639bb031) Reviewed by: rikard Tested in sqldev, 2013-06-17 23:45:46, OK | Insert Column, New attribute for customer Tested in sqltest, 2013-06-18 12:10:01, OK | Insert Column, New attribute for customer Tested in sqlprod , 2013-09-14 20:57:44, OK | Insert Column, New attribute for customer
Implementation: Run script using Jenkins deploy server
Implementation & Review: Executed by Jenkins/CI-Server, 2013-09-14 20:57 Scripts successfully applied
Advanced TopicsSQL Output - real life example
Tips and tricks - wrong is right
One of the most difficult challenges is to overcome “wrong is right” principle
● When a changeset has been committed, it can not be modified even if it is “wrong”
● You must write another changeset to correct the previous one
● The “wrong” changeset will hit your production database followed by the “right” changeset
Tips and tricks - stored procedure
If you have stored procedures the should obviously be versioned controlled
But the CRC does not fit well as that requires the stored procedure to be repeated for every modification
The “run always” attribute saves the day. This will make sure that whenever there is a change it is applied -- a better fit for stored procedures
Tips and tricks - stored procedures
--liquibase formatted sql--changeset rikard:1 runOnChange:true
ALTER procedure javaforum…
● Above we use a plain sql file with two special comments that magically makes it a changeset
Demo
Stored procedure
1. Update database with plain SQL file2. Change procedure and update again..
Tips and Tricks - neutralizing data
● The “run always” attribute can also be used to neutralize data in dev and test environments
● Useful in cases when the database is copied for production to other environments
● The changes that has been applied is stored in the database itself, therefore it is safe to copy clone a database
Tips and Tricks - neutralizing data
<changeSet id="1" author="joe" runAlways="true" context="sqld,sqlt,localhost"> <comment>Change users psw on (localhost, dev, test) to 'Secret' and their email to '[email protected]'</comment> <update tableName="user"> <column name="email" value="[email protected]"/> <column name="password" value="Secret"/> <where>email != '[email protected]' OR password != 'Secret'</where> </update></changeSet>
Real life experience
We have used Liquibase in projects for many years, more than 50 man years of development with very little problems
Overall score 97%by Roger and Rikard
Alternatives
● Flyway● c5-db-migration● dbdeploy● mybatis● MIGRATEdb● migrate4j● dbmaintain● AutoPatch
Q & A
