Upload
marlon-pierce
View
58
Download
0
Embed Size (px)
Citation preview
Welcome to the SGCI Webinar!• We will be starting shortly.
• Your audio has been muted, and you are encouraged to turn off your video during the presentation.• Controls for these are near the bottom of
the right-side control panel for BlueJeans.
• You may submit questions at any time using Chat, and the moderator will share them with the presenter when appropriate.
• This presentation will be recorded and slides will be posted.
1
Creating a developer pipeline by teaching
gateway technologiesMarlon Pierce, Suresh Marru
Science Gateways Research Center, Indiana [email protected], [email protected]
http://sgrc.iu.edu
The Pipeline Problem
Finding and retaining people to work on science gateways is tough.
If only there was an untapped reservoir of talent close by....
The Pipeline Solution
Recruit students to work with us by teaching a graduate level class on the underpinnings of gateways.
Embrace change.
What Is a Science Gateway?A Science Gateway is a user-centric environment
for conducting online computational science
(Not actual IU students)
What if Mark Zuckerberg Read More Chemistry and Biology Books?
Goal 1: Apply basic distributed computing concepts to Science
Gateways.
Goal 2: Apply new architectures, methodologies, and technologies to
Science Gateways: Microservices, DevOps
Browser
Web Interface Server
Application Server
Server SDK
Client SDK
IU: Big Red 2
Resource Plugins
XSEDE: Stampede XSEDE: Comet Juelich: Jureca
HTTPS
HTTP or TCP/IP
Facebook doesn’t look like this, so why should science gateways?
Goal 3: Teach open source software practices
Challenges for Science Gateways• Providing a rich user experience • Defining an API for the application server• Defining the right microservices.• Implementing the components, wiring them together correctly.• Supporting multiple gateway tenants• Fault tolerance for components• State management• Continuous delivery• Security management• Supporting full scientific exploratory cycle
What Do We Teach?
A quick survey of course material
Micro-Services
DevOpsDistributed Systems
Putting It All Together: Gateways and Microservices
Virtualization, Containers, Docker
API and Metadata Model Design
Messaging Systems: RabbitMQ, Apache Kafka
Resource Management and Scheduling: Mesos, Aurora, and Torque
Distributed State Management: Zookeeper, Consul
Security: OAuth2 and OpenIDConnect
Putting It All In Place: Continuous Integration and Deployment
Mission Accomplished! Students learned on their own.
More Information and Next Steps
• http://courses.airavata.org• Additional material in Canvas
• Details on assignments• We are interested in taking this to a broader audience
Thank you for participating!
• Please offer your feedback with our 30-second evaluation:http://sciencegateways.org/webinareval
• Join us next month (March 8) for
Building a Modern Research Data Portal with Globus – Introduction to the Globus Platform
36
Science!
Spring 2016 Course Overview• Students organized into teams of 3-4• 8 biweekly project milestones, each depended on the previous
assignment.• Basic sequence, Part 1:
• Learn to submit jobs on IU’s Karst Cluster, MOAB scheduler• Develop core job management application• Design an API• Design a Web front end• Capture job execution metadata
• All code must be in GitHub, must build cleanly on the grader’s laptop
Spring 2016 Project Milestones, Part 2• Integrate with a continuous integration and deployment system
• Travis-CI and Amazon Code-Deploy• Apache Jenkins on Jetstream
• Break up your monolithic Web applications into microservices• Combine with CI/CD
Spring 2016: What Worked, What Didn’t?• Small class, we got to know the
students pretty well.• Students managed to build basic
functioning gateways that used CI/CD automatically linked to GitHub accounts.
• Students got bogged down in job submission issues.
• Students had trouble breaking up their monolithic Web applications into microservices.
Fall 2016: Begin at the End• For Fall 2016, we reverse the order.• Students learned CI/CD and microservices in the first assignments.• We moved aggressively to problems of load balancing, fault tolerance,
and containerization of services.• We de-emphasized job submission to traditional clusters
• Used Apache Mesos and Apache Aurora for this.
Fall 2016: What Worked, What Didn’t• All the students were able to
keep up with the assignments.• Many were using technologies for
the first time.
• Some surprises• Some teams took it as a challenge
to use different approaches.
• Didn’t have the chance to get to know students as well as before.
• Larger class, graders
• Monthly assignments were too large.
• Biweekly worked better.
Spring 2017: Advanced Topics• In the Spring 2016 and Fall 2016 courses, we knew the answers
(mostly).• Now in Spring 2017 course, we tackle problems we don’t know the
answers to.• Or, if you prefer, for a given problem,
• Introductory Class: Find an answer that works• Advanced Class: Find the best answer
• How do you do that?