1

Data accessibility, performance and security for a hosted Primo

• Robert Watt (Service Development Manager,

Information Resources, Library Services, King’s College London)

• email: robert.watt@kcl.ac.uk

2

Agenda

• Brief introduction to King’s

• Background to the Primo implementation project at King’s

• Accessibility: requirements; test methodology; outcomes and resolution

• Performance: requirements; test methodology; outcomes and resolution

• Security: requirements; test methodology; outcomes and resolution

• Lessons learned

• Questions

3

Brief introduction to King’s

• 5 campuses around London

• 21,500 FTEs

• 9 Schools

• Collections: >855,000 print books; >30,000 e journals; >338,000

e books and >700 databases

• Ex Libris products used: Aleph; bX; Metalib; Primo; SFX; Ustat

4

Background to the Primo implementation project at King’s

• Tendered for a discovery tool in mid 2012

• Key strength of Primo was its integration with Aleph

• Primo implementation project kicked off October 2012

• Go live date initially December 2012 – optimistic!

• King’s rigorous testing identified problems around accessibility; performance; security - reported to EXL in December/January 2012/2013 – go live postponed

• King’s worked with EXL first half 2013 to resolve go live issues

• Primo (branded “Library Search”) went live on 29/05/2013

5

King’s Primo

• librarysearch.kcl.ac.uk

6

Accessibility: King’s requirements for a cloud service• King’s places great importance on accessibility for its users: the

College’s Strategic Plan specifically mentions “addressing accessibility issues”

• King’s complies with the Equality Act 2010, including its provisions on disability

• King’s requires end-user and back-office web interfaces should conform with current World Wide Web Consortium (W3C) Web Content Accessibility Guidelines (WCAG) to Level A

• Primo is required to work with King’s College London accessibility tools including Dragon naturally speaking software and Supernova screen reading software

7

Accessibility testing: methods and tools used

• King’s tested the Primo web pages against WCAG guidelines with Firefox’s HTML validator add on; Achecker software and Total Validator

• Primo tested with the following accessibility tools:

Dragon Naturally Speaking 11.5 speech recognition software

SuperNova 11.54 screen reading software

8

Accessibility testing: outcomes for web page validators• Verification that the web interface complies with the WCAG standards resulted in 1,500 errors i.e.

malformed html and css– Achecker– Input validator (Total Validator – the same software that EXL use)

• Tests conducted using King’s accessibility software by accessibility expert– Full interface revealed some deficiencies – Mobile interface seemed to work better

• AChecker – identified thousands of problems (pivotal details)

9

How did King’s work with Ex Libris to resolve the issues around accessibility? (1)

• Issues were discussed in a number of conference calls between King’s and EXL’s security experts

• EXL service pack 4.4 to address many of the issues identified by King’s note: King’s to verify how many have been resolved

• EXL were validating the wrong pages i.e. the PDS log in page rather than the actual Primo pages

• EXL said that they did not identify the accessibility issues discovered by King’s, chiefly because: “they resulted from a change in the HTML that was not visible in the user interface, and was therefore not checked the way that changes to the user interface are typically checked. We have amended our accessibility procedures so that this will not be repeated”

 

10

How did King’s work with Ex Libris to resolve the issues around accessibility? (2)

• EXL provided King’s with a written commitment to meet the accessibility requirements expressed in King’s tender document for Primo

• King’s have created an accessibility link in the FE to a mobile version of the interface. Because the mobile version is less verbose it makes it faster when used with a screen reader.

 

11

Performance: King’s requirements for a cloud service• King’s Primo tender stipulated that response times should be under 5

seconds to retrieve and display results from the single Primo index

• King’s identified that the performance of Primo in testing prior to go live vastly exceeded the 5 seconds threshold, and with only a handful of users!

• King’s didn’t specify the amount of load in Primo regarding the number of searches per minute on the tender requirement

• King’s expected the load to be higher than the Aleph load of 140 – 170 peak “load”

12

13

Performance testing: methods and tools used

• King’s performed load testing of Primo with Apache Jmeter to: Execute a search and go to the results Access the user’s Library Account

• The load testing was conducted with an incremental increase of the number of users mimicking use of Primo

• Methodology– Multiple locations

– Search expressions extracted from Aleph Apache log - “160,000”

14

Performance testing outcomes (1)

• On first test run system brought to a halt*– Due to not closing the sessions

• On second run PDS server brought to a halt– Script amended to login and logout to close session.

• On third run and later runs– Script amended to do searches in a loop without opening new sessions

– Search loops paused for one minute before proceeding

– Number of simultaneous users gradually increased until system brought a halt

• Final run was successful up to 200 user searches per minute

* : more than a 30 second response time

15

A note on how server sizing is performed by Ex Libris

Number Users* = FTE/200

King’s sizing = 20,000 users/ 200 = 100 users (normal usage)

* A user is defined as doing one search per minute

16

How did King’s work with Ex Libris to resolve the issues around performance?

• EXL made configuration changes on the production server which resolved the performance issues: the major initial cause being zombie Apache processes that over loaded the system, then minor adjustments to the application server’s environment

• EXL and King’s negotiated a formal commitment from EXL to resize the King’s Primo server after go live, should it be required i.e. if it exceeded predicted usage of 200 searches per minute and caused performance issues

• King’s redrafted their SLA with Ex Libris to specify different categories of performance requirements e.g. a P1 rated category (not available to multiple users) has a resolution upper limit of 2 working days

• Note: EXL said this was the first time they’d set specific response targets for any customer in this way

17

How do King’s measure performance on an ongoing basis

• King’s uses a centralised monitoring platform called Nagios– Verifies the system is up by calling the xml server and searching for a particular title

– Uses Jmeter’s script to verify 5 seconds delay in search requests• Primo• BBC (external web page for comparison

• Future improvement– Supply library with periodic performance report

18

Example of Nagios report

19

Security: King’s requirements for a cloud service

• King’s Primo tender stipulated that Primo “must comply with all relevant data protection and data security law and best practice, including the UK Data Protection Act, and preferably be stored on servers in the European Economic Area (EEA)”

• King’s requires the encryption of the sign in pages for Primo FE; and the back office for production and staging

• King’s requires that personal or confidential information travel through an encrypted (HTTPS) tunnel. This includes the personal data in the FE and the OvP interface with ALEPH server

• King’s requires permission from the vendor to run penetration (PEN) tests on any cloud based system it subscribes to

20

Security: methods and tools used

• King’s used several Penetration software applications to assess the security compliance of Primo. All these tools were used with Vanilla configuration and executed using black-box PEN testing. No Gray PENetration methods were used. Some of the tools used were: Burp Scanner and Nessus Vulnerability Scanner

21

Security testing outcomes

• PENetration tests– Revealed serious XSS reflection* issues

• King’s security policies– All user personal information needs to travel in secure encrypted channels (HTTPS)

– All login screens need to travel through secure encrypted channels (HTTPS)

* XSS reflection – is an attack where a user can inject javascript on the end-users browser

22

How did King’s work with Ex Libris to resolve the issues around security? (1)

• EXL escalated resolution of security issues to their senior staff and held a number of conference calls with King’s and EXL security experts

• Shlomo Sanders (EXL Chief Technology Officer) visited King’s in June.

• EXL implemented service pack (4.3) to address security fix, which resolved King’s original issues

• King’s then re-ran the PEN tests and uncovered new security issues

23

How did King’s work with Ex Libris to resolve the issues around security? (2)

• EXL installed a local hot fix to address these latest security issues

• King’s reported that the hot fix had not fixed these security issues

• EXL installed another hot fix which did resolve the outstanding issues

• EXL enabled HTTPS on all Primo web pages (note: King’s wanted just the sign in page encrypted) - this caused problems on BO staging server as King’s lost access to it for one month!

• OVP encryption of data transfer – Ex Libris are developing this and plan to release in a service pack in early 2014

24

Lessons learned: Communication

• Be frank and open in exchanges with the vendor

• Direct communication between experts on both the customer and vendor side yields results

• EXL demonstrated flexibility and a willingness to adapt their procedures to address King’s requirements as a result of fruitful communication between King’s and EXL

25

Lessons learned: Project and implementation

• When drawing up project tender requirements, ensure that all requirements are included. These include an up-to-date figure for FTEs to ensure that the Primo server is sized appropriately

• Be transparent at the tender stage about how you intend to test Primo. Be explicit about how you intend to test the system

• Ensure SLAs are clearly and unambiguously defined before the contract is signed, particularly around timescales for support where there are problems, and performance

• Ensure all requirements for accessibility; performance; security are defined at the tender stage and at contractual level (before anything is signed)

• Specify in the tender the specific (particularly third party) software that Primo is required to work with e.g. Dragon

• Consider the option of service credits to compensate for any failure in SLAs

26

Lessons learned: Testing

• Clarify the test parameters before testing i.e. exactly what is being tested and which tools will be used by both customer and vendor to ensure that results are comparable

• Where tools differ, coordinate testing to ensure both vendor and customer are testing exactly the same data with the same outcomes

• Conduct rigorous testing internally on cloud based systems – don’t assume that things are ok (just because the vendor says so)

• Ensure that the service is monitored regularly for performance issues with monitoring tools.

27

Questions?

?