Copyright © 2011 Flatirons Solutions Corporation
1
MarkLogic Production Deployments
February 15, 2011
How to make them clean/repeatable/testable
Copyright © 2011 Flatirons Solutions Corporation
2
Who Am I?
Brad Rix [email protected] Lead for Content Technology Practice
Flatirons Solutions CorporationFlatirons Solutions is a Colorado based system integrator and long-time MarkLogic partner specializing in XML publishing, dynamic content delivery, and digital asset management for both commercial and government clients.
Copyright © 2011 Flatirons Solutions Corporation
3
Why Do I Care About This Stuff?
Problem: You’ve written some cool XQuery It does great things that will help your business/customer But oops: the more you write code, the harder it gets:
New developers break code they don’t understand Deployments to new environment don’t work and the issues are
hard to resolve More code being written causes more instabilities which causes
longer debug/deploy cycles Which results in you spending less time writing cool XQuery
and more time fixing the problems described above
Answer: Invest some time and engineering to create a
build/test/deploy/continuous-integration infrastructure which will greatly improve the ratio of (time doing fun stuff/time doing un-fun stuff)
Copyright © 2011 Flatirons Solutions Corporation
4
Agenda
General Goals
Automatic Deployments
Automatic Configuration
Automated Unit Testing
Hudson Automated Builds
Lessons Learned
Copyright © 2011 Flatirons Solutions Corporation
5
General Deployment Goals
Repeatable Procedures Ability to automate deployment of XQuery
modules Ability to automate building of environments
to replicate exact configuration among dev/test/production environments
Ability to run unit tests against the code & environment
Automate running of unit tests
Copyright © 2011 Flatirons Solutions Corporation
6
Automated Deployments
XQuery modules stored in database Used ant task to deploy modules Use Docs HTTP server or create a single HTTP
server that has a single module stored that manages deployment
Ant task zips up files and POST them to the HTTP server on the Docs database where the simple XQuery module is deployed.
Setting collections/permissions/etc on the files as they are ingested.
Copyright © 2011 Flatirons Solutions Corporation
7
Automated Deployments (ant task)
First zip modules into a single zip file
<target name="package“> <mkdir dir="${dist.home}"/> <zip destfile="${dist.home}/${ml.
name}.zip"> <fileset dir="${build.home}/src/${ml.
name}"/> </zip> </target>
Copyright © 2011 Flatirons Solutions Corporation
8
Automated Deployment (ant task)
Properties set in build.properties configurable per environment
Post zip file to preloaded endpoint in Docs database
<target name=“deploy"> <exec executable="wget" dir="${dist.home}"
failonerror="true" vmlauncher="true"> <arg value="--user=${ml.ingest.user}"/> <arg value="--password=${ml.ingest.password}"/> <arg value="--header=Content-Type:application/zip"/> <arg value="--post-file=${ml.name}.zip"/> <arg
value="${ml.db.docs.host}/admin/loadModulesDB.xqy?modulesdb=${ml.modules.db}&serverroot=/"/>
</exec> </target>
Copyright © 2011 Flatirons Solutions Corporation
9
Automated Deployment (module)
Modules parse zip file to get file (snippet):
for $zip-input in xdmp:zip-manifest($xquery-modules-zip)//*:part
return
Get Module: xdmp:zip-get($xquery-modules-zip, $filename, <options xmlns="xdmp:zip-get"> </options>)Then load module into database:xdmp:document-insert($uri, $doc,
$permissions)
Copyright © 2011 Flatirons Solutions Corporation
10
Automated Configuration
Goal is to be able to configure all database configurations including Application servers/application settings/database settings/range indexes/security roles/etc.
Create a configuration file that contains all of the required information.
Allow the configuration to change and able to re-run on new or existing environment
Run the configuration via ant task or xquery module
Copyright © 2011 Flatirons Solutions Corporation
11
Automated Database Configuration
Configuration file per environment
Partial Configuration of database<database name=“MyDBName”>
<word-positions>true</word-positions><element-value-positions>true</element-value-positions> <element-range-indexes> <index> <scalar-type>string</scalar-type>
<namespace-uri>http://mhhe.com/meta/resolved</namespace-uri>
<local-name>type</local-name> <collation>http://marklogic.com/collation/</collation> <range-value-position>false</range-value-position> </index>
…..
Copyright © 2011 Flatirons Solutions Corporation
12
Automated Configuration (ant task)
Ant task to call a module to configure the server.
Post the file to an endpoint or have the module load the configuration from a file.
<target name="setupdatabase"> <runxquery
module="admin/configureEnv.xqy?source=filesystem&uri=${ml.home.url}/Docs/config/environment-config.xml&env=${ml.environment}&restart=true"
host="${ml.db.docs.host}" user="${ml.user}" password="${ml.password}"/>
</target>
Copyright © 2011 Flatirons Solutions Corporation
13
Configuration Module
Configuration module will read the configuration file
Read the XML options and run the appropriate admin services API functions to configure the server/database.
Optionally restart the server depending on parameters that require restart to change
Copyright © 2011 Flatirons Solutions Corporation
14
Sample Function to Create Indexes
let $database-name := $database-config/@name let $dbid := xdmp:database($database-name) (: remove old ones first :) let $cluster-config := admin:database-delete-range-element-index($cluster-config,$dbid, admin:database-get-range-element-indexes ($cluster-config, $dbid)) let $retv := for $index in $database-config/element-range-indexes/index let $rangespec := admin:database-range-element-index($index/scalar-type,
$index/namespace-uri, $index/local-name, $index/collation, $index/range-value-position ) return try { xdmp:set($cluster-config, admin:database-add-range-element-index($cluster-
config, $dbid, $rangespec) ) } catch ($err) { if ($err/error:code eq "ADMIN-DUPLICATEITEM") then () else error(xs:QName("ERROR"), xs:string($err/error:message)) }
return $cluster-config
Copyright © 2011 Flatirons Solutions Corporation
15
XQuery Unit Testing
Know that all existing modules work as expected
Changes in modules still work for previous expected behavior
At Flatirons we developed a java XQueryUnit that utilizes XML Unit for differencing XML files
Framework uses ant to run all unit tests Incorporated into entire build process so that
the deploy target can run the tests to ensure that everything works properly in this environment
Copyright © 2011 Flatirons Solutions Corporation
16
Unit Tests (configuration file)
Call a module and expect known return If XML does not match return, then test will fail
<xqu:test xqu:name="HelloWorldInline"> <xqu:xqy_test> <xqu:xqy_test_file xqu:uri="modules/util/testHelloWorld.xqy" /> </xqu:xqy_test> <xqu:expected_result> <xqu:xml_result> <xqu:xml_result_src> <hello> world </hello> </xqu:xml_result_src> </xqu:xml_result> </xqu:expected_result> </xqu:test>
Copyright © 2011 Flatirons Solutions Corporation
17
Unit Tests – Testing Restful Endpoint
To test an http restful call into MarkLogic, write an XQuery wrapper that performs the xdmp:http-get (or post).
Return the data back from the module and compare that to expected output in the configuration file.
Copyright © 2011 Flatirons Solutions Corporation
18
Run Unit Tests
Run ant test
test: [junit] Testsuite:
com.mhhe.xmlunit.UnitTestSuite [junit] Tests run: 5, Failures: 0, Errors: 0,
Time elapsed: 3.078 sec [junit] [junit] Testcase: HelloWorldInline took 0.453
sec [junit] Testcase: HelloWorldFile took 0.016
sec [junit] Testcase: TestCreateProject took 2.109
sec [junit] Testcase: TestFavorites took 0.172 sec [junit] Testcase: TestGetAsset took 0.219 sec
Copyright © 2011 Flatirons Solutions Corporation
19
Unit Tests (Failed Test)
[junit] [junit] Testcase: HelloWorldInline took 0.313 sec [junit] FAILED [junit] Unexpected error received during test
'HelloWorldInline': junit.framework.AssertionFailedError: org.custommo
nkey.xmlunit.Diff [junit] [different] Expected text value 'world' but was
' [junit] failed_test [junit] ' - comparing <hello
...>world</hello> at /hello[1]/text()[1] to <hello ...> [junit] failed_test [junit] </hello> at /hello[1]/text()[1]
Copyright © 2011 Flatirons Solutions Corporation
20
Automated Build Testing (Hudson)
Continuous build testing with Hudson (http://hudson-ci.org/)
Hudson is a self contained web application (just install and run). Includes Jetty web server as part of deployment.
Run builds on a schedule (hourly) as each developer changes code.
Build will run full configuration/ code deploy/ unit tests at each interval
If any failure then e-mail user who checked in code (along with buildmeister)
Quick demo of Hudson now….
Copyright © 2011 Flatirons Solutions Corporation
21
Lessons Learned
Tightly type the input and output of all modules This will detect when function is called with empty or
wrong parameters quickly.
Disable Function Mapping (will attempt to map calls to other functions) declare option xdmp:mapping "false";
Unit Test as much as possible