Synergy 2014 - Syn122 Moving Australian National Research into the Cloud

SYN122Moving Australian national research into the cloud

Nick Golovachenko

Session Abstract

To increase the productivity of their researchers, Australian universities are leveraging cloud architectures to deliver on-demand access to resources with unprecedented agility and efficiency. This session highlights NeCTAR, a Super Science project whose lead agent is the University of Melbourne.

National Servers Project (NSP) provides a robust cloud platform for hosting core NeCTAR services for the Australian eResearch community that require high levels of reliability and availability.

Our Goal

To provide a reliable, safe and secure environment for Australian researchers to host their important research applications which means they don’t have to worry about anything except managing their research.

Moving Australian research into the cloud

Moving Australian research into the cloud

Introduction

Session Summary

1. Background 2. What is NeCTAR3. What is eResearch4. What is the NSP5. How was it Built and

Implemented6. Challenges7. Questions

Background – Melbourne

Background – Melbourne

The University of Melbourne

The University of Melbourne

What is NeCTAR

• The University of Melbourne was commissioned by the then Department of Industry, Innovation, Science, Research and Tertiary Education (DIISR) deliver of the NeCTAR program.

• The National eResearch Collaboration Tools and Resources (NeCTAR) Program aims to enhance research collaboration and research outcomes by providing Information and Communication Technology (ICT) infrastructure that:

What is NeCTAR

• Creates new information centric research capabilities;

• Significantly simplifies the combining of instruments, data, computing, and analysis applications; and

• Enables the development of research workflows based on access to multiple resources.

What NeCTAR has built

NeCTAR is delivering to Australian researchers:

• Virtual Laboratories that connect a range of resources of relevance to a specific research community:

• eResearch Tools that support problem-oriented eResearch needs and further the achievement of improved research outcomes;

• A federated Research Cloud Built for virtual machines that only run from minutes to months with not so strict entry, security and maintenance criteria

• A National Server offering a secure and robust hosting service with strict entry, security and maintenance criteria, Built for virtual machines that need to run continuously for years

What is eResearch

• What is eResearch: 

• The term e-Research (alternately spelled eResearch) refers to the use of information technology to support existing and new forms of research. E-research to other disciplines, including the humanities and social sciences.

• Examples of e-Research problems range across disciplines and include:

• modelling of ecosystems or economies• exploration of human genome structures• studies of large linguistic corpuses• integrated social policy analyses

• The main features of e-Research are that it:

• is collaborative• uses grid computing technologies• is data intensive• E-Research includes research activities that use a spectrum of advanced information and communication

technology (ICT) capabilities. It embraces new research methodologies emerging from increasing access to:

What is the NSP

• The National Servers Program is a cloud based Infrastructure as a Service (IaaS) service which hosts critical e-Research services for the Australian research community. Uniquely Australian, the NSP provides a robust national network of virtual servers and platforms to support many eResearch services such as the operations for the Research Cloud, virtual laboratories, tools and data.

• The NSP is currently free for Australian researchers. It provides a safe and secure environment for Australian researchers to host their important research application which means they don’t have to worry about anything except managing their research.

What is NSP

Allocation Review Process

Applications to host services on the National Servers Program (NSP) are reviewed by the NSP Allocation Committee. This committee consists of representatives from around Australia, and they review applications according to the following four criteria:

• The service underpins the integrated management or access to collaboration, data, compute or analysis resources and capabilities,

• The service is considered a core to other eResearch services and activities,

• The service will support activities by researchers from multiple institutions, and

• The service has suitable maintenance and user support resources in place.

How it was Built

Research Community Consultation • Requirements gathering Demand projections Architecture development

Provisioning

• Basic Access PhaseManual provisioning of VM Limited backup Limited high availability

• VMs =10 to 50 at each site

Full Service Phase• Built-in high availability and Self Service • VMs =250 to 500 at each site

Full Service Phase Features

• The NSP provides an unmanaged service for maximum flexibility.

• Advanced Self service features allow Researchers to control all stages of their VM lifecycle.

• Researchers can build their applications to leverage features such as failover and load balancing between the two zones (Data Centres). Researchers can utilise a Portal for Self Service account management within their allocated Research Group, they can also manage snapshots, templates and advance networking, e.g., Firewall, NAT, DNS, DHCP, VPN and load balancing.

NSP How it was Built and Implemented

Zone OneQueensberry Street (QS2) Data Centre

Zone TwoNoble Park (NP)

Data Centre

100 KM

Hypervisor Clusters

Primary Storage

Secondary Storage

NFS storage with cross site replication

Production VMsProduction VMsStaging VMs

NSP Management Stack

• CloudPortal

• CloudPlatform (Citrix supported version of Apache CloudStack)

• OpenLDAP - for user Authentication

• MySQL – underlying database for both Portal and Platform

• XenServer Hypervisors currently being used for the Management Stack plus Virtual Machines

Snapshots, Templates & Volumes

Supported Operating Systems & template offerings• Centos 6.x• Ubuntu 10.x & 12.04• Debian 6.x & 7.x• Windows Server

Volumes can be either the standard root OS volume or an additional data volume:We can offer additional volumes:• Small – 10GB• Medium – 20GB• Large – 40GB

Who is using the NSP

Who is using the NSP

IMOS is designed to be a fully-integrated national array of observing equipment to monitor the open oceans and coastal marine environment around Australia, covering physical, chemical and biological variables.

IMOS – Use Case

Quadrant – Project Management Software for Researchers

Quadrant is a cloud-based project management and data collection

software tool that allows participant-based researchers to work

collaboratively and efficiently from a self-managed centralised site.

Terrestrial Ecosystem Research Network (TERN)

The Terrestrial Ecosystem Research Network (TERN) connects ecosystem scientists and enables them to collect, contribute, store, share and integrate data across disciplines. Collectively this increases the capacity of the Australian ecosystem science community to advance science and contribute to effective management and sustainable use of our ecosystems.

TERN is delivering critical research infrastructure and supporting national and international networks of scientists, environmental managers and stakeholders, needed to improve understanding and management of Australia’s ecosystems.

Challenges

• Initially NSP was provisioned on Shared Storage – Impacted by Performance and Reliability.

• Researchers always wanted largest VM Types and Storage Offerings.

• Confusion with the two NeCTAR Clouds.• Transition to operation has had it’s challenges.• Sustainability of the service still needs to be

completed.• Cloudplatform Disaster Recovery - Promised• Citrix Support – Xenserver and Cloud teams where

two team not one, resulting in separate support requests for one issue

Conclusion

• The National Server Program (NSP) was established to provide a hosting home for Australian eResearch Services.

• Any publicly funded eResearch service provider may apply through a merit scheme to have their service hosted on the NSP.

• The services hosted on the NSP are used by researchers across Australian and Internationally.

Questions

Over to you…