Upload
eric-ritchie
View
673
Download
3
Tags:
Embed Size (px)
DESCRIPTION
Jenkins (to name just one CI tool) is a central pillar of agile development and does a terrific job of automating the project build process. Providing a reliable deployment path to a cluster of servers is something that it does less well and forms the challenge being faced by many today. In this talk Novadex GmbH, a leader in cloud based customer communication management, will demonstrate their pragmatic solution to this challenge which uses the toolkit provided by Zend Server
Citation preview
Novadex GmbH 2013. All rights reserved, also regarding any disposal, exploita>on, reproduc>on, edi>ng, distribu>on, as well as in the event of applica>ons for industrial property rights.
Jenkins entwined with deployment and pragma>sm
Eric Ritchie DevOps Engineer Novadex +49 (0) 7142 / 788905-‐0 [email protected]
Novadex GmbH 2013. All rights reserved, also regarding any disposal, exploita>on, reproduc>on, edi>ng, distribu>on, as well as in the event of applica>ons for industrial property rights.
Eric Ritchie bids you welcome!
20/11/13 2
§ currently working as DevOps Engineer and System Administrator at Novadex GmbH
§ a Zend Framework and PHP ZCE & MySQL DBA
§ a Linux System Administrator with 18 years experience
§ a PHP developer since version 3.0.18 (or so)
§ known to speak at a conference or two
Never heard of him? Aside from looking like this, he is:
Novadex GmbH 2013. All rights reserved, also regarding any disposal, exploita>on, reproduc>on, edi>ng, distribu>on, as well as in the event of applica>ons for industrial property rights.
LeaerMaschine: Custom-‐tailored Communica>on
Personalized data stream Cross-‐Channel Output
Collabora>on
Portal Op>on Campaign management
CRM integra>on
Content management E-‐commerce integra>on
Data protec>on Security
20/11/13 3
Novadex GmbH 2013. All rights reserved, also regarding any disposal, exploita>on, reproduc>on, edi>ng, distribu>on, as well as in the event of applica>ons for industrial property rights.
What I read in a book once…
20/11/13 4
„DevOPs should aim for 10 deployments per day“
Novadex GmbH 2013. All rights reserved, also regarding any disposal, exploita>on, reproduc>on, edi>ng, distribu>on, as well as in the event of applica>ons for industrial property rights.
Novadex’s key tool: Jenkins
20/11/13 5
§ Great for Con?nuous Integra?on
§ Can hide complex processes behind a nice buDon
§ Deployment remains one of those complex processes § Requires a lot of scrip>ng/manual work § This risks our deployment goal
Novadex GmbH 2013. All rights reserved, also regarding any disposal, exploita>on, reproduc>on, edi>ng, distribu>on, as well as in the event of applica>ons for industrial property rights.
An example of that scrip>ng
20/11/13 6
<target name="package"> <echo message="...using UPSTREAM_BUILD_NUMBER received from triggering job for packagenaming. UPSTREAM_BUILD_NUMBER is [${UPSTREAM_BUILD_NUMBER}]"/> <!-‐-‐ leaermaschine-‐complete packaging -‐-‐> <!-‐-‐ wipe the dir from possible previous copy opera>on -‐-‐> <delete includeemptydirs="true"> <fileset dir="${package.contents.basedir}" includes="**/*"/> <fileset dir="${package.basedir}/DEBIAN" includes="**/*"/> </delete> <!-‐-‐ prepare debian control file -‐-‐> <copy todir="${package.basedir}/DEBIAN" overwrite="true"> <fileset file="resource/package/test/control"/> <filterset begintoken="[[" endtoken="]]"> <filter token="version" value="${UPSTREAM_BUILD_NUMBER}-‐1"/> <filter token="name" value="leaermaschine-‐complete"/> <filter token="depends" value=""/> <filter token="descrip>on" value="Complete Leaermaschine applica>on"/> </filterset> </copy> <!-‐-‐ copy pos>nstalla>on script -‐-‐> <copy file="${basedir}/resource/package/test/pos>nst_lm" tofile="${package.basedir}/DEBIAN/pos>nst"/> <chmod file="${package.basedir}/DEBIAN/pos>nst" perm="775"/> <copy todir="${package.contents.basedir}"> <fileset dir="${export.basedir}"> <snip> </fileset> </copy> <exec executable="sudo" dir="${working.basedir}" failonerror="true"> <arg line="sudo dpkg -‐b ${package.basedir} ${dist.basedir}/leaermaschine-‐complete_${UPSTREAM_BUILD_NUMBER}-‐1_all.deb"/> </exec> </target>
Novadex GmbH 2013. All rights reserved, also regarding any disposal, exploita>on, reproduc>on, edi>ng, distribu>on, as well as in the event of applica>ons for industrial property rights.
Deployment v1.x?
20/11/13 7
§ Applica?on bundled into a .deb package § As shown on the previous slide (yeah, that’s what it did) § Shell scrip>ng available for in deployment manipula>ons
§ SCP’d to target machine, installed via SSH exec § Need to set up SSL cer>ficates to allow password free access
Novadex GmbH 2013. All rights reserved, also regarding any disposal, exploita>on, reproduc>on, edi>ng, distribu>on, as well as in the event of applica>ons for industrial property rights.
Problems?
20/11/13 8
§ Who is comfortable with bash shell scrip?ng?
§ What happens when something goes wrong? § Ever tried to roll back a copied .deb package in a hurry?
§ Not really all that cluster friendly § Ok, we have Chef/Puppet if we like overkill
§ Disk cluDer § Each version of the .deb package is cached
§ Need to grant unlimited access to live servers, argh! § Remember that SSL cer>ficate we installed?
§ What if we need to support Red Hat also? § Hypothe>cal in the case of Novadex, but s>ll a valid problem.
Novadex GmbH 2013. All rights reserved, also regarding any disposal, exploita>on, reproduc>on, edi>ng, distribu>on, as well as in the event of applica>ons for industrial property rights.
Enter Zend Server deployment (and me)
20/11/13 9
Cluster/Cloud ready, plaxorm agnos>c, PHP scrip>ng and version handling with integrated roll back.
Yes, I used to work for Zend ;)
Novadex GmbH 2013. All rights reserved, also regarding any disposal, exploita>on, reproduc>on, edi>ng, distribu>on, as well as in the event of applica>ons for industrial property rights.
Tale of the liale known tool: zs-‐manage
20/11/13 10
§ We all have heard about the Web API, right? § You may be forgiven for thinking it is a new thing
§ We have had command line deployment tools since day one § But nobody knew about them, which is a pity
§ zs-‐manage has many advantages § Can leverage all deployment features § Does not rely on SSL cer>ficates for security § Same commands for local/cluster/cloud deployment
§ But beware, there are some caveats § Pragma>sm to follow...
Novadex GmbH 2013. All rights reserved, also regarding any disposal, exploita>on, reproduc>on, edi>ng, distribu>on, as well as in the event of applica>ons for industrial property rights.
Let’s empower Jenkins…
20/11/13 11
<!-‐-‐ retrieve the app.id for the applica>on, specified by app.name. may be empty if app hasn't already deployed -‐-‐> <exec executable="bash" dir="${basedir}" outputproperty="app.id" errorproperty="err" failonerror="true"> <arg value="-‐c" /> <arg value="/usr/local/zend/bin/zs-‐manage app-‐get-‐status -‐N ${zs.api.name} -‐U ${zs.url} -‐K ${zs.api.key} 2>&1 | grep "APPINFO\\s.*${app.name}.*" | awk '{print $2}'" /> </exec> <if> <equals arg1="${app.id}" arg2="" /> <then> <echo message="Couldn't determine app.id for app.name '${app.name}' -‐ preparing ini>al deployment of applica>on ..." /> <!-‐-‐ deploy applica>on via zend server command -‐-‐> <exec executable="/usr/local/zend/bin/zs-‐manage" dir="/tmp" failonerror="true"> <arg line="app-‐deploy -‐u "NDX_ENV=${zs.ndx.env}" -‐c -‐b hap://${app.name}/ -‐a ${app.name} -‐p ${zs.package.des>na>ondir}/${zs.package.name} -‐N ${zs.api.name} -‐K ${zs.api.key} -‐U ${zs.url}" /> </exec> </then> <else> <echo message="app.name '${app.name}', app.id is ${app.id} -‐ preparing the update of applica>on ..." /> <!-‐-‐ update applica>on via zend server command -‐-‐> <exec executable="/usr/local/zend/bin/zs-‐manage" dir="/tmp" failonerror="true"> <arg line="app-‐update -‐u "NDX_ENV=${zs.ndx.env}" -‐i ${app.id} -‐p ${zs.package.des>na>ondir}/${zs.package.name} -‐N ${zs.api.name} -‐K ${zs.api.key} -‐U ${zs.url}" /> </exec> </else> </if>
Novadex GmbH 2013. All rights reserved, also regarding any disposal, exploita>on, reproduc>on, edi>ng, distribu>on, as well as in the event of applica>ons for industrial property rights.
Crea>ng the package to be deployed
20/11/13 12
§ We have a choice § Zend Studio -‐ a manual process § zdpack -‐ Zend standard § zip -‐ needs no introduc>on
§ Novadex have used zip at ?mes § But currently use the following code
<exec executable="/usr/local/zend/bin/zdpack" dir="${working.basedir}" failonerror="true"> <!-‐-‐ NOTE: name of the created package is the name specified in deployment.xml name-‐tag -‐-‐> <arg line="pack -‐-‐src-‐dir=${package.contents.basedir} -‐-‐scripts-‐dir=${basedir}/resource/zs_deploy_new/scripts -‐-‐package-‐descriptor=${working.basedir}/deployment.xml -‐-‐output-‐dir=${working.basedir}" /> </exec> <!-‐-‐ rename created package to reflect version of package contents -‐-‐> <move file="${working.basedir}/${app.name}.zpk" tofile="${working.basedir}/${zs.package.name}" overwrite="true"/> <!-‐-‐ copy the new package from temp loca>on to package directory -‐-‐> <copy file="${working.basedir}/${zs.package.name}" todir="${zs.package.des>na>ondir}" overwrite="true"/>
Novadex GmbH 2013. All rights reserved, also regarding any disposal, exploita>on, reproduc>on, edi>ng, distribu>on, as well as in the event of applica>ons for industrial property rights.
Sadly, size does maaer
20/11/13 13
§ You don’t leave your doors unlocked when you leave home § Most admins are equally careful with their servers
§ Web API could be an aDack vector § Which is why Zend take steps to secure it § Web API calls must be signed with, amongst other informa>on, a >mestamp
§ To help prevent replay aDacks, calls must complete within 300 seconds § This security feature can shoot us in the foot § In short, the network speed determines our maximum packet size
Novadex GmbH 2013. All rights reserved, also regarding any disposal, exploita>on, reproduc>on, edi>ng, distribu>on, as well as in the event of applica>ons for industrial property rights.
Working around the size issue
20/11/13 14
§ No perfect solu?ons, yet § I am working on Zend ;)
§ But we can apply some pragma?sm § Package can be broken into sub-‐packages § Zend Deployment has a new Library feature, use it § Pull instead of push § Upgrade the pipe § Time travel § Or, emulate the GUI
Novadex GmbH 2013. All rights reserved, also regarding any disposal, exploita>on, reproduc>on, edi>ng, distribu>on, as well as in the event of applica>ons for industrial property rights.
Our solu>on
20/11/13 15
<echo message="copying ${zs.package.name} to ${remote.hostname}" /> <scp file="${zs.package.des>na>ondir}/${zs.package.name}" todir="${remote.user}@${remote.hostname}:${remote.dir}" keyfile="${remote.keyfile}" /> <echo message="app.name '${app.name}', app.id is '${app.id}' -‐ preparing the update of applica>on ..." /> <!-‐-‐ execute zs-‐manage locally on host where package should be deployed! -‐-‐> <sshexec host="${remote.hostname}" username="${remote.user}" keyfile="${remote.keyfile}" command="/usr/local/zend/bin/zs-‐manage app-‐update -‐u "NDX_ENV=${zs.ndx.env}" -‐i ${app.id} -‐p ${remote.dir}/${zs.package.name} -‐N ${zs.api.name} -‐K ${zs.api.key} -‐U ${remote.webapi.url}" />
GUI emula?on: It works for us
Just remember: SCP/SSH may not be allowed by your opera>ons team. You may need another solu>on ! !
Novadex GmbH 2013. All rights reserved, also regarding any disposal, exploita>on, reproduc>on, edi>ng, distribu>on, as well as in the event of applica>ons for industrial property rights.
Memory and lots of it
20/11/13 16
§ Ques?on: Where does all this deployment data go? § Hint: The WebAPI is part of the GUI
§ The GUI allocates memory equivalent to three ?mes the packet size § Since the data is loaded, copied and finally unpacked (don’t ask)
§ Where is the GUI’s memory limit configured? § Not in the GUI, that would spoil the fun ;) § Can you guess?
Answer: The memory_limit variable in: /usr/local/zend/gui/lighapd/etc/php-‐fcgi.ini
Novadex GmbH 2013. All rights reserved, also regarding any disposal, exploita>on, reproduc>on, edi>ng, distribu>on, as well as in the event of applica>ons for industrial property rights.
Aaack of the cluster
20/11/13 17
§ Zend deployment can be hard to synchronise § In other words, you cannot guarantee when all nodes will run the
same version of the applica>on
§ An extreme case is the “run once” node § Database migra>ons are usually performed here, so deployment is slower
§ This caused problems with our “In maintenance” page § Time for some extreme pragma>sm…
Novadex GmbH 2013. All rights reserved, also regarding any disposal, exploita>on, reproduc>on, edi>ng, distribu>on, as well as in the event of applica>ons for industrial property rights.
I am a Job Queue fan… so I wrote this
20/11/13 18
// Sanity check if (!class_exists('ZendJobQueue')){ logger("Job Queue seems to be disabled!"); exit(1); } if (getenv("ZS_RUN_ONCE_NODE") == 1) { $name = DEPLOYMENT_JOB_BASENAME.$appCurrentVersion; $queue = new ZendJobQueue(); // Before scheduling our job, just make sure that it has not been scheduled before $total = null; $jobArray = $queue-‐>getJobsList(array('name' => $name), $total); if ($total != 0) { foreach($jobArray AS $job) { $queue-‐>removeJob($job['id']); } } // Schedule the job script to run a long >me in the future (since we don't really need it to run) $runAt = date('Y-‐m-‐d H:i:s', strto>me('+7 days')); $semaphoreJobId = $queue-‐>createHapJob('hap://www.google.de/search?q=dummy', array(), array('name' => $name, 'schedule_?me' => $runAt)); if (!$semaphoreJobId) { logger("Semphore job was not accepted in the queue!"); exit(1); } }
Novadex GmbH 2013. All rights reserved, also regarding any disposal, exploita>on, reproduc>on, edi>ng, distribu>on, as well as in the event of applica>ons for industrial property rights.
…and this at the end of the script
20/11/13 19
if (getenv("ZS_RUN_ONCE_NODE") == 1) { // We are the RUN ONCE script so clear the semaphore $queue-‐>removeJob($semaphoreJobId); } else { // We are not the RUN ONCE script so wait for it to clear the semaphore $queue = new ZendJobQueue(); $total = 1; $status = ZendJobQueue::JOB_STATUS_PENDING | ZendJobQueue::JOB_STATUS_SCHEDULED; while ($total != 0) { $jobArray = $queue-‐>getJobsList(array('name' => $name, 'status' => $status), $total); sleep(1); } }
We now plan to use an Nginx, that spontaneously appeared in our architecture, to set our maintenance page. Nevertheless, it was s>ll a cool idea ;)
Novadex GmbH 2013. All rights reserved, also regarding any disposal, exploita>on, reproduc>on, edi>ng, distribu>on, as well as in the event of applica>ons for industrial property rights.
Finding out what is happening
20/11/13 20
§ So far so good, but we s?ll don’t have true integra?on § Jenkins has no liale* visibility over the deployment process
§ We can log the output of our deployment scripts § But these are not available to Jenkins § We can make them available to the GUI, at least, by adding the paths to the
GUI_AVAILABLE_LOGS table in the Zend Server database § More convenient, but we s>ll need to go check the GUI a�er each deployment
* Last minute slide hack
Novadex GmbH 2013. All rights reserved, also regarding any disposal, exploita>on, reproduc>on, edi>ng, distribu>on, as well as in the event of applica>ons for industrial property rights.
WebAPI to the rescue… again
20/11/13 21
§ Features added to the WebAPI allow registered logs to be remotely read § logsReadLines § logsGetLogfile
§ We can finally integrate log output into our Jenkins reports
§ Just a few problems § Finding the “run once” node § These methods are not implemented in zs-‐manage (yet) § Wri>ng the code (I haven’t had the >me yet)
Novadex GmbH 2013. All rights reserved, also regarding any disposal, exploita>on, reproduc>on, edi>ng, distribu>on, as well as in the event of applica>ons for industrial property rights.
www.novadex.com
www.leaermaschine.com
Please scan this …and tell me where it goes
Honestly, marke>ng wouldn’t tell me!
Thank you very much for your aaen>on
20/11/13 22
Eric Ritchie DevOps Engineer hDps://joind.in/9926
+49 (0) 7142 / 788905-‐0 [email protected]
Novadex GmbH Stuagarter Straße 59 74321 Bie>gheim-‐Bissingen Germany