MarkLogic Production Deployments

Copyright © 2011 Flatirons Solutions Corporation 1

MarkLogic Production Deployments

April 7, 2011

How to make them clean/repeatable/testable


Who Am I?

Brad Rix [email protected] Lead for Content Technology Practice

Flatirons Solutions CorporationFlatirons Solutions is a Colorado based system integrator and long-time MarkLogic partner specializing in XML publishing, dynamic content delivery, and digital asset management for both commercial and government clients.


Why Do I Care About This Stuff?

Problem: You’ve written some cool XQuery It does great things that will help your business/customer But oops: the more you write code, the harder it gets:

New developers break code they don’t understand Deployments to new environment don’t work and the issues are

hard to resolve More code being written causes more instabilities which causes

longer debug/deploy cycles Which results in you spending less time writing cool XQuery

and more time fixing the problems described above Answer:

Invest some time and engineering to create a build/test/deploy/continuous-integration infrastructure which will greatly improve the ratio of (time doing fun stuff/time doing un-fun stuff)


General Deployment Goals

Repeatable Procedures Ability to automate deployment of XQuery

modules Ability to automate building of environments

to replicate exact configuration among dev/test/production environments

Ability to run unit tests against the code & environment

Automate running of unit tests


Automated Deployments

XQuery modules stored in database Used ant task to deploy modules Use Docs HTTP server or create a single HTTP

server that has a single module stored that manages deployment

Ant task zips up files and POST them to the HTTP server on the Docs database where the simple XQuery module is deployed.

Setting collections/permissions/etc on the files as they are ingested.


Automated Deployments (ant task)

First zip modules into a single zip file

<target name="package“> <mkdir dir="${dist.home}"/> <zip destfile="${dist.home}/${ml.

name}.zip"> <fileset dir="${build.home}/src/${ml.

name}"/> </zip> </target>


Automated Deployment (ant task)

Properties set in build.properties configurable per environment

Post zip file to preloaded endpoint in Docs database

<target name=“deploy"> <exec executable="wget" dir="${dist.home}"

failonerror="true" vmlauncher="true"> <arg value="--user=${ml.ingest.user}"/> <arg value="--password=${ml.ingest.password}"/> <arg value="--header=Content-Type:application/zip"/> <arg value="--post-file=${ml.name}.zip"/> <arg

value="${ml.db.docs.host}/admin/loadModulesDB.xqy?modulesdb=${ml.modules.db}&serverroot=/"/>

</exec> </target>


Automated Deployment (module)

Modules parse zip file to get file (snippet):

for $zip-input in xdmp:zip-manifest($xquery-modules-zip)//*:part

return

Get Module: xdmp:zip-get($xquery-modules-zip, $filename, <options xmlns="xdmp:zip-get"> </options>)Then load module into database:xdmp:document-insert($uri, $doc, $permissions)


Automated Configuration

Goal is to be able to configure all database configurations including Application servers/application settings/database settings/range indexes/security roles/etc.

Create a configuration file that contains all of the required information.

Allow the configuration to change and able to re-run on new or existing environment

Run the configuration via ant task or xquery module


Automated Database Configuration

Configuration file per environment

Partial Configuration of database<database name=“MyDBName”>

<word-positions>true</word-positions><element-value-positions>true</element-value-positions> <element-range-indexes> <index> <scalar-type>string</scalar-type>

<namespace-uri>http://mhhe.com/meta/resolved</namespace-uri> <local-name>type</local-name> <collation>http://marklogic.com/collation/</collation> <range-value-position>false</range-value-position> </index>

…..


Automated Configuration (ant task)

Ant task to call a module to configure the server.

Post the file to an endpoint or have the module load the configuration from a file.

<target name="setupdatabase"> <runxquery

module="admin/configureEnv.xqy?source=filesystem&uri=${ml.home.url}/Docs/config/environment-config.xml&env=${ml.environment}&restart=true"

host="${ml.db.docs.host}" user="${ml.user}" password="${ml.password}"/>

</target>


Configuration Module

Configuration module will read the configuration file

Read the XML options and run the appropriate admin services API functions to configure the server/database.

Optionally restart the server depending on parameters that require restart to change


Sample Function to Create Indexes

let $database-name := $database-config/@name let $dbid := xdmp:database($database-name) (: remove old ones first :) let $cluster-config := admin:database-delete-range-element-index($cluster-config,$dbid, admin:database-get-range-element-indexes ($cluster-config, $dbid)) let $retv := for $index in $database-config/element-range-indexes/index let $rangespec := admin:database-range-element-index($index/scalar-type,

$index/namespace-uri, $index/local-name, $index/collation, $index/range-value-position ) return try { xdmp:set($cluster-config, admin:database-add-range-element-index($cluster-config,

$dbid, $rangespec) ) } catch ($err) { if ($err/error:code eq "ADMIN-DUPLICATEITEM") then () else error(xs:QName("ERROR"), xs:string($err/error:message)) }

return $cluster-config


XQuery Unit Testing

Know that all existing modules work as expected Changes in modules still work for previous expected

behavior At Flatirons we developed a java XQueryUnit that

utilizes XML Unit for differencing XML files Can be done using HtmlUnit/XMLUnit/Performance Meters

Framework uses ant to run all unit tests Incorporated into entire build process so that the

deploy target can run the tests to ensure that everything works properly in this environment

To test an http restful call into MarkLogic, write an XQuery wrapper that performs the xdmp:http-get (or post).

Return the data back from the module and compare that to expected output in the configuration file.


Unit Tests (configuration file)

Call a module and expect known return If XML does not match return, then test will fail

<xqu:test xqu:name="HelloWorldInline"> <xqu:xqy_test> <xqu:xqy_test_file xqu:uri="modules/util/testHelloWorld.xqy" /> </xqu:xqy_test> <xqu:expected_result> <xqu:xml_result> <xqu:xml_result_src> <hello> world </hello> </xqu:xml_result_src> </xqu:xml_result> </xqu:expected_result> </xqu:test>


Run Unit Tests

Run ant test

test: [junit] Testsuite:

com.mhhe.xmlunit.UnitTestSuite [junit] Tests run: 5, Failures: 0, Errors: 0,

Time elapsed: 3.078 sec [junit] [junit] Testcase: HelloWorldInline took 0.453

sec [junit] Testcase: HelloWorldFile took 0.016

sec [junit] Testcase: TestCreateProject took 2.109

sec [junit] Testcase: TestFavorites took 0.172 sec [junit] Testcase: TestGetAsset took 0.219 sec


Unit Tests (Failed Test)

[junit] [junit] Testcase: HelloWorldInline took 0.313 sec [junit] FAILED [junit] Unexpected error received during test

'HelloWorldInline': junit.framework.AssertionFailedError: org.custommo

nkey.xmlunit.Diff [junit] [different] Expected text value 'world' but was ' [junit] failed_test [junit] ' - comparing <hello

...>world</hello> at /hello[1]/text()[1] to <hello ...> [junit] failed_test [junit] </hello> at /hello[1]/text()[1]


Automated Build Testing (Hudson)

Continuous build testing with Hudson (http://hudson-ci.org/) or Jenkins (http://jenkins-ci.org/)

Hudson is a self contained web application (just install and run). Includes Jetty web server as part of deployment.

Run builds on a schedule (hourly) as each developer changes code or manual ad-hoc deployments from a web-interface

Build will run full configuration/ code deploy/ unit tests at each interval. Can automate deployments to multiple environments.

If any failure then e-mail user who checked in code (along with buildmeister)

http://hudson-ci.org/

http://hudson-ci.org/

http://jenkins-ci.org/

http://jenkins-ci.org/


Dashboard (live)

Shows current builds Allows kicking off new builds Can see information on success

and failed builds


Console Output (live)


Lessons Learned

Tightly type the input and output of all modules This will detect when function is called with empty or

wrong parameters quickly. Disable Function Mapping

declare option xdmp:mapping "false"; Unit Test as much as possible

Documents

MarkLogic Production Deployments