Automate that

Embed Size (px)

Citation preview

Page 1: Automate that
Page 2: Automate that


Has this happened to you?• Email to users results in

50+ undeliverable

• Need to verify the users in Active Directory

• Then “deactivate” former employees in Crowd

• 750 mouse clicks later, you’re done! http://www.flickr.com/photos/left-hand/4231405740/

Page 3: Automate that

Scripting Atlassian applications with Python

Dave ThomasConfiguration Management Architect, FIS


Automate That!

Page 4: Automate that



• Use cases for scripting

• Atlassian APIs available for scripting

• The awesome power and simplicity of python

• Examples

Page 5: Automate that


When is scripting useful?

• Automate time consuming tasks

• Perform data analysis

• Cross-reference data from multiple systems

Page 6: Automate that


Some specific use cases

• Crowd – Deactivate Users and remove from all groups

• Bamboo – Disable all plans in a project

• JIRA – Release Notes

• Subversion – custom commit acceptance

• Custom build processes – pull code linked to a specific issue into a patch archive

Page 7: Automate that

Why Scripts?Why Not Plugins?


• I’m not a Java Developer

• Installing new plugins can require a restart

• Prefer to minimize ad hoc changes on the server

• Need to correlate information from several systems

• Need an agile process to accommodate changing requirements

Page 8: Automate that


APIs for scripting(that we avoid if possible)

• The user interface• Can do anything a user can do• Reporting tasks are relatively easy (particularly when xml is available)• Actions are relatively hard (and prone to breakage)• Capture browser traffic with livehttpheaders, firebug, etc• Form token checking can be an obstacle

• XML-RPC and SOAP• Relatively low-level interface• Many actions available• Relatively complex to use

Page 9: Automate that


More APIs for scripting(the ones we prefer to use)

• RESTful Remote APIs (now deprecated)• High level interface• Supports a handful of actions

• Now emerging: “real” REST interfaces• High level interface• Supports a handful of actions• http://confluence.atlassian.com/display/REST/Guidelines+for+Atlassi


Page 10: Automate that


Why Python?• Powerful standard libraries

• Http(s) with cookie handling• XML and JSON• Unicode

• Third Party Libraries• SOAP• REST• Templates• Subversion

• Portable, cross-platform

Page 11: Automate that


Python Versions

• 2.x• Ships with most linux distributions• Lots of third-party packages available

• 3.x• Latest version• Deliberately incompatible with 2.x• Not as many third-party libraries

Page 12: Automate that


HTTP(s) with Python• Python 2

• httplib – low level, all HTTP verbs• urllib – GET and POST, utilities• urllib2 – GET and POST using Request class, easier manipulation

of headers, handlers for cookies, proxies, etc.• Python 3

• http.client – low level, all HTTP verbs• http.parse - utilities• urllib.request – similar to urllib2

• Third-Party• httplib2 – high-level interface with all HTTP verbs, plus caching,

compression, etc.

Page 13: Automate that


Example 1JIRA Issue Query & Retrieval

Page 14: Automate that


Discovering URLs for XML

Page 15: Automate that


Simple Issue Retrievalimport urllib, httplibimport xml.etree.ElementTree as etree

jira_serverurl = 'http://jira.atlassian.com'jira_userid = 'myuserid'jira_password = 'mypassword'

detailsURL = jira_serverurl + \"/si/jira.issueviews:issue-xml/JRA-9/JRA-9.xml" + \"?os_username=" + jira_userid + "&os_password=" + jira_password

f = urllib.urlopen(detailsURL)tree=etree.parse(f)f.close()

Construct a URL that looks like the one in the UI, with extra parms for

our user auth

Open the URL with one line!Parse the XML with one


Page 16: Automate that


Find details in XML

details = tree.getroot()print "Issue: " + details.find("channel/item/key").textprint "Status: " + details.find("channel/item/status").textprint "Summary: " + details.find("channel/item/summary").textprint "Description: " + details.find("channel/item/description").text

Issue: JRA-9Status: OpenSummary: User Preference: User Time ZonesDescription: <p>Add time zones to user profile. That way the dates displayed to a user are always contiguous with their local time zone, rather than the server's time zone.</p>

Find based on tag name or path to


Page 17: Automate that


Behind the scenes…cookies!httplib.HTTPConnection.debuglevel = 1f = urllib.urlopen(detailsURL)

send: 'GET /si/jira.issueviews:issue-xml/JRA-9/JRA-9.xml?os_username=myuserid&os_password=mypassword HTTP/1.0\r\nHost: jira.atlassian.com\r\nUser-Agent: Python-urllib/1.17\r\n\r\n'reply: 'HTTP/1.1 200 OK\r\n'header: Date: Wed, 20 Apr 2011 12:04:37 GMTheader: Server: Apache-Coyote/1.1header: X-AREQUESTID: 424x2804517x1header: X-Seraph-LoginReason: OKheader: X-AUSERNAME: myuseridheader: X-ASESSIONID: 19b3b8oheader: Content-Type: text/xml;charset=UTF-8header: Set-Cookie: JSESSIONID=A1357C4805B1345356404A65333436D3; Path=/header: Set-Cookie: atlassian.xsrf.token=AKVY-YUFR-9LM7-97AB|e5545d754a98ea0e54f8434fde36326fb340e8b7|lin; Path=/header: Connection: close

Turn on debugging and see exactly what’s


JSESSIONID cookie sent from JIRA

Page 18: Automate that


Authentication• User credentials determine:

• The data returned• The operations allowed

• Methods Available:• Basic Authentication• JSESSIONID Cookie• Token Method

Page 19: Automate that


Basic Authentication

• Authentication credentials passed with each request

• Can be used with REST API

Page 20: Automate that


JSESSIONID Cookie• Authentication credentials passed once;

then cookie is used

• Used when scripting the user interface

• Can be used with REST API for JIRA, Confluence, and Bamboo

Page 21: Automate that


Token Method• Authentication credentials passed once;

then token is used

• Used with Fisheye/Crucible REST

• Used with Deprecated Bamboo Remote API

Page 22: Automate that


Obtaining a cookie

• Scripting the user interface login page

• Adding parameters to the user interface URL: “?os_username=myUserID&os_password=myPassword”

• Using the JIRA REST API

Page 23: Automate that


JIRA REST Authenticationimport urllib, urllib2, cookielib, json

# set up cookiejar for handling URLscookiejar = cookielib.CookieJar()myopener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cookiejar))

creds = { "username" : jira_userid, "password" : jira_password }queryurl = jira_serverurl + "/rest/auth/latest/session"req = urllib2.Request(queryurl)req.add_data(json.dumps(creds))req.add_header("Content-type", "application/json")req.add_header("Accept", "application/json")fp = myopener.open(req) fp.close()

urllib2 handles cookies automatically. We just need to

give it a CookieJar

Request and response are both

JSONWe don’t care about response, just the


Page 24: Automate that


Submitting a JIRA Querywith the user interface

# Search using JQLqueryJQL = urllib.quote("key in watchedIssues()")queryURL = jira_serverurl + \ "/sr/jira.issueviews:searchrequest-xml/temp/SearchRequest.xml" + \ "?tempMax=1000&jqlQuery=" + queryJQLfp = myopener.open(queryURL)

# Search using an existing filterfilterId = "20124"queryURL = jira_serverurl + \ "/sr/jira.issueviews:searchrequest-xml/" + \ "{0}/SearchRequest-{0}.xml?tempMax=1000".format(filterId)fp = myopener.open(queryURL)

Pass any JQL Query

Or Pass the ID of an existing shared filter

Page 25: Automate that


A JQL Query using REST# Search using JQLqueryJQL = "key in watchedIssues()"IssuesQuery = { "jql" : queryJQL, "startAt" : 0, "maxResults" : 1000 }queryURL = jira_serverurl + "/rest/api/latest/search"req = urllib2.Request(queryURL)req.add_data(json.dumps(IssuesQuery))req.add_header("Content-type", "application/json")req.add_header("Accept", "application/json")fp = myopener.open(req)data = json.load(fp)fp.close()

Pass any JQL Query

Request and response are both


Page 26: Automate that


XML returned from user interface query

An RSS Feed with all issues and requested

fields that have values

Page 27: Automate that


JSON returnedfrom a REST query

{u'total': 83, u'startAt': 0, u'issues': [{u'self': u'http://jira.atlassian.com/rest/api/latest/issue/JRA-23969',

u'key': u'JRA-23969'}, {u'self': u'http://jira.atlassian.com/rest/api/latest/issue/JRA-23138',

u'key': u'JRA-23138'}, {u'self': u'http://jira.atlassian.com/rest/api/latest/issue/BAM-2770',

u'key': u'BAM-2770'}, {u'self': u'http://jira.atlassian.com/rest/api/latest/issue/BAM-2489',

u'key': u'BAM-2489'}, {u'self': u'http://jira.atlassian.com/rest/api/latest/issue/BAM-1410',

u'key': u'BAM-1410'}, {u'self': u'http://jira.atlassian.com/rest/api/latest/issue/BAM-1143',

u'key': u'BAM-1143'}], u'maxResults': 200}

A list of the issues found, with links to

retrieve more information

Page 28: Automate that


JSON issue details

All applicable fields are returned, even if

there’s no value

Expand the html property to get rendered html for description,


Page 29: Automate that


What’s the difference?<reporter username="mlassau">Mark Lassau [Atlassian]</reporter><customfield id="customfield_10160" key="com.atlassian.jira.toolkit:dayslastcommented">

<customfieldname>Last commented</customfieldname><customfieldvalues>

1 week ago</customfieldvalues>


u'reporter': {u'type': u'com.opensymphony.user.User', u'name': u'reporter', u'value': {

u'self': u'http://jira.atlassian.com/rest/api/latest/user?username=mlassau', u'displayName': u'Mark Lassau [Atlassian]', u'name': u'mlassau'}},

u'customfield_10160': {u'type': u'com.atlassian.jira.toolkit:dayslastcommented', u'name': u'Last commented', u'value': 604800},

XML values are display strings

REST values are type-dependent

Page 30: Automate that

REST vs. non-REST



• More roundtrips to query JIRA and get issue details

• Returns all fields

• Values require type-specific interpretation

• Easier to transition issues

• Easier to get info for projects, components


• Can query based on existing filter

• XML returns only fields that contain values

• Values always one or more display strings

• Can do anything a user can do (with a little work)

Page 31: Automate that


Example 2Cross-referencing JIRA, Fisheye, and

Bamboo build results

Page 32: Automate that


Which build resolved my issue?• Bamboo keeps track of “related issues” (based on issue IDs

included in commit comments), but doesn’t know when issues are resolved.

• If we know the issue is resolved in JIRA, we can look to see the latest build that lists our ID as a “related issue”

• Not a continuous integration build? We’ll need to look in fisheye to determine the highest revision related to this issue and then look in bamboo to see if a build using this revision has completed successfully.

Page 33: Automate that


To Fisheye for related commits!

queryURL = FisheyeServer + "/rest-service-fe/changeset-v1/listChangesets" + \ "?rep={0}&comment={1}&expand=changesets".format(FisheyeRepo, myissue)

req = urllib2.Request(queryURL)auth_string = '{0}:{1}'.format(fisheye_userid,fisheye_password)base64string = base64.encodestring(auth_string)[:-1]req.add_header("Authorization", "Basic {0}".format(base64string))response = myopener.open(req)issuecommits=etree.parse(response).getroot()response.close()

Query a specific fisheye repository for a commit with our JIRA issue ID in the


Use basic auth headers to


Page 34: Automate that


Fisheye changesets returned<results expand="changesets"> <changesets> <changeset>

<csid>130948</csid> <date>2011-04-29T12:35:56.150-04:00</date> <author>lc6081</author> <branch>trunk</branch> <comment>MYJIRAPROJECT-2823 Modified to add parameters</comment> <revisions size="1" />

</changeset> </changesets></results>

Page 35: Automate that


Parsing the changesetscommits = []for changeset in issuecommits.findall("changesets/changeset"): commits.append(changeset.findtext("csid"))

commits.sort()print "Highest commit is: " + commits[-1]

Highest commit is: 130948

Page 36: Automate that


Logging into Bamboo


cookiejar = cookielib.CookieJar()myopener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cookiejar))

queryURL = bambooServer + "/userlogin!default.action“

params = urllib.urlencode({ "os_username" : bambooUserid, "os_password" : bambooPassword})response = myopener.open(queryURL, params) response.close()

Using a POST to the user interface login screen to

retrieve a JSESSIONID cookie

Page 37: Automate that


Querying for build results

# Warning: This is a very resource-intensive operation. # You should consider limiting the number of builds returnedqueryURL = bambooServer + "/rest/api/latest/result/MYPROJECT-MYPLAN" + \ "?expand=results[-10:-1].result.jiraIssues"

req = urllib2.Request(queryURL)req.add_header("Accept", "application/xml")response = myopener.open(req)results=etree.parse(response).getroot()response.close()

Use negative indexes to return the last entries in build list, e.g. [-10:-1] returns last ten builds in list

Request the related issues

Ask for XML(JSON also available)

Page 38: Automate that


Example (partial) build results

<results expand="results"><link href="http://mybamboo.domain.com:8080/rest/api/latest/result/MYPROJECT-MYPLAN" rel="self" /><results expand="result" max-result="25" size="46" start-index="0"><result expand="comments,labels,jiraIssues,stages" id="3146125" key="MYPROJECT-MYPLAN-26" lifeCycleState="Finished" number="26" state="Successful">

<link href="http://mybamboo.domain.com:8080/rest/api/latest/result/MYPROJECT-MYPLAN-26" rel="self" /><buildStartedTime>2011-04-29T05:04:14.460-05:00</buildStartedTime><buildCompletedTime>2011-04-29T05:34:35.687-05:00</buildCompletedTime><buildRelativeTime>4 days ago</buildRelativeTime><vcsRevisionKey>4483</vcsRevisionKey><buildReason>Code has changed</buildReason><comments max-result="0" size="0" start-index="0" /><labels max-result="0" size="0" start-index="0" /><jiraIssues max-result="1" size="1" start-index="0">

<issue iconUrl="http://myjira.domain.com/images/icons/bug.gif" issueType="Defect" key="MYJIRAPROJECT-1629" summary="Need to display an error message when balance is zero."><url href="http://myjira.domain.com/browse/MYJIRAPROJECT-1629" rel="self" />

</issue></jiraIssues><stages max-result="1" size="1" start-index="0" />


Can also expand comments, labels, and


jiraIssues property has been expanded here

Page 39: Automate that


Walking through build resultsfor result in results.findall("results/result"): print result.get("key") + ":" print "\tRevision: " + result.findtext("vcsRevisionKey") issues = [issue.get("key") for issue in result.findall("jiraIssues/issue")] print "\tIssues: " + ", ".join(issues)

MYPROJECT-MYPLAN-31: Revision: 4489 Issues: MYJIRAPROJECT-1658MYPROJECT-MYPLAN-30: Revision: 4486 Issues: MYJIRAPROJECT-1630MYPROJECT-MYPLAN-29: Revision: 4485 Issues: MYJIRAPROJECT-1616, MYJIRAPROJECT-1663

Page 40: Automate that


Example 3Removing a user from a Crowd group

Page 41: Automate that


Beyond GET and POST

connection = httplib.HTTPConnection('myCrowdServer.mydomain.com:8080')operation = 'DELETE'urlpath = "/rest/usermanagement/latest/user/group/direct" + \ "?username={0}&groupname={1}".format(userToRemove, fromGroup)body = Noneauth_string = '{0}:{1}'.format(crowdAppName, crowdAppPassword)base64string = base64.encodestring(auth_string)[:-1]headers = {'Authorization' : "Basic {0}".format(base64string)}connection.request(operation, urlpath, body, headers)response = connection.getresponse()print response.status, response.reasonconnection.close()

204 - group membership is successfully deleted 403 - not allowed to delete the group membership 404 - the user or group or membership could not be found

Lower-level HTTPConnection needed

Authenticate as a Crowd


Page 42: Automate that


A few loose ends

• Be prepared to handle Unicode strings

• Error handling – not shown here, but important!

• Formatting output – several python libraries for handling templates are available

• REST Interfaces – you can write your own!http://confluence.atlassian.com/display/DEVNET/Plugin+Tutorial+-+Writing+REST+Services

Page 44: Automate that