72
THE RESEARCH SOFTWARE ENGINEER Building careers, collaborations, groups and communities @alysbrett

SOFTWARE ENGINEER THE RESEARCH€¦ · experiment team make decisions Open-source tool from neutron scattering Plugin & interface for fusion data 9. Data Access: Simple Access Layer

  • Upload
    others

  • View
    1

  • Download
    0

Embed Size (px)

Citation preview

Page 1: SOFTWARE ENGINEER THE RESEARCH€¦ · experiment team make decisions Open-source tool from neutron scattering Plugin & interface for fusion data 9. Data Access: Simple Access Layer

THE RESEARCH SOFTWARE ENGINEER

Building careers, collaborations, groups and communities

@alysbrett

Page 2: SOFTWARE ENGINEER THE RESEARCH€¦ · experiment team make decisions Open-source tool from neutron scattering Plugin & interface for fusion data 9. Data Access: Simple Access Layer

Research software at UKAEA

2

Page 3: SOFTWARE ENGINEER THE RESEARCH€¦ · experiment team make decisions Open-source tool from neutron scattering Plugin & interface for fusion data 9. Data Access: Simple Access Layer

The UK’s national fusion lab

Fusion energy research and related technology

3

Page 4: SOFTWARE ENGINEER THE RESEARCH€¦ · experiment team make decisions Open-source tool from neutron scattering Plugin & interface for fusion data 9. Data Access: Simple Access Layer

Fusion energy research at UKAEA

Two Tokamaks experiments

4

JET MAST-U

Page 5: SOFTWARE ENGINEER THE RESEARCH€¦ · experiment team make decisions Open-source tool from neutron scattering Plugin & interface for fusion data 9. Data Access: Simple Access Layer

Technology facilities at UKAEA

5

▸ Robotics▸ Materials▸ Apprentice training

£86 million new government funding:National Fusion Technology Platform

Page 6: SOFTWARE ENGINEER THE RESEARCH€¦ · experiment team make decisions Open-source tool from neutron scattering Plugin & interface for fusion data 9. Data Access: Simple Access Layer

UKAEA: Software Engineering Group

Develop & maintain data systems and tools

Help researchers with software

6

Page 7: SOFTWARE ENGINEER THE RESEARCH€¦ · experiment team make decisions Open-source tool from neutron scattering Plugin & interface for fusion data 9. Data Access: Simple Access Layer

JET data pipeline

Process Access Analyse VisualiseStore Discover

7

Page 8: SOFTWARE ENGINEER THE RESEARCH€¦ · experiment team make decisions Open-source tool from neutron scattering Plugin & interface for fusion data 9. Data Access: Simple Access Layer

Data Dashboard

8

Discovering data

At-a-glance summary

Microservices + CMS front end

Page 9: SOFTWARE ENGINEER THE RESEARCH€¦ · experiment team make decisions Open-source tool from neutron scattering Plugin & interface for fusion data 9. Data Access: Simple Access Layer

Control-room visualisation

Tool for viewing real time data

Customised views help experiment team make decisions

Open-source tool from neutron scattering

Plugin & interface for fusion data

9

Page 10: SOFTWARE ENGINEER THE RESEARCH€¦ · experiment team make decisions Open-source tool from neutron scattering Plugin & interface for fusion data 9. Data Access: Simple Access Layer

Data Access: Simple Access Layer

10

▸ Diverse data

▸ Varied, complex access methods

▸ Unfamiliar scientists

▸ Long term value

Page 11: SOFTWARE ENGINEER THE RESEARCH€¦ · experiment team make decisions Open-source tool from neutron scattering Plugin & interface for fusion data 9. Data Access: Simple Access Layer

Signal data Modelling LogsImagesVideoAudio

SALREST API

Storage layer

Client

11

Data Access: Simple Access Layer

Machine meta-data

Page 12: SOFTWARE ENGINEER THE RESEARCH€¦ · experiment team make decisions Open-source tool from neutron scattering Plugin & interface for fusion data 9. Data Access: Simple Access Layer

Programming at UKAEA

159 survey responses

549 Gitlab users

12

Page 13: SOFTWARE ENGINEER THE RESEARCH€¦ · experiment team make decisions Open-source tool from neutron scattering Plugin & interface for fusion data 9. Data Access: Simple Access Layer

The research software problem

13

Page 14: SOFTWARE ENGINEER THE RESEARCH€¦ · experiment team make decisions Open-source tool from neutron scattering Plugin & interface for fusion data 9. Data Access: Simple Access Layer

Study at: https://goo.gl/gmHwGm 14

Page 15: SOFTWARE ENGINEER THE RESEARCH€¦ · experiment team make decisions Open-source tool from neutron scattering Plugin & interface for fusion data 9. Data Access: Simple Access Layer

31 repositories ~600,000 papers

15

⅔ of papers indicate

software reliance in 2017

Page 16: SOFTWARE ENGINEER THE RESEARCH€¦ · experiment team make decisions Open-source tool from neutron scattering Plugin & interface for fusion data 9. Data Access: Simple Access Layer

SOFTWARE IS ESSENTIAL FOR RESEARCH

16

Page 17: SOFTWARE ENGINEER THE RESEARCH€¦ · experiment team make decisions Open-source tool from neutron scattering Plugin & interface for fusion data 9. Data Access: Simple Access Layer

Who writes research software?

17

Study of main UK academic jobs site

Page 18: SOFTWARE ENGINEER THE RESEARCH€¦ · experiment team make decisions Open-source tool from neutron scattering Plugin & interface for fusion data 9. Data Access: Simple Access Layer

18

Page 19: SOFTWARE ENGINEER THE RESEARCH€¦ · experiment team make decisions Open-source tool from neutron scattering Plugin & interface for fusion data 9. Data Access: Simple Access Layer

Missing role

Researcher career structures don’t reward software focus

Software Engineer

ResearcherResearcher Developer

19

Page 20: SOFTWARE ENGINEER THE RESEARCH€¦ · experiment team make decisions Open-source tool from neutron scattering Plugin & interface for fusion data 9. Data Access: Simple Access Layer

The “Research Software Engineer”

Software engineering skills + understanding of research

Software Engineer

ResearcherResearcher Developer

20

Page 21: SOFTWARE ENGINEER THE RESEARCH€¦ · experiment team make decisions Open-source tool from neutron scattering Plugin & interface for fusion data 9. Data Access: Simple Access Layer

A note about names and categories

▸ A common name is powerful

▸ Pre-existing roles and teams

▸ Job titles and group names vary

eg “engineer” is protected term

▸ Community is inclusive

21

Page 22: SOFTWARE ENGINEER THE RESEARCH€¦ · experiment team make decisions Open-source tool from neutron scattering Plugin & interface for fusion data 9. Data Access: Simple Access Layer

Career case study: Dr Louise Brown

PhD Mech. Eng. Extensive software engineering experience (academia and industry)

Started working on TexGen textiles software package at University of Nottingham

22

Page 23: SOFTWARE ENGINEER THE RESEARCH€¦ · experiment team make decisions Open-source tool from neutron scattering Plugin & interface for fusion data 9. Data Access: Simple Access Layer

As a “Research Fellow” Louise couldn’t get career progression

Because metrics are based on research publications

23

Page 24: SOFTWARE ENGINEER THE RESEARCH€¦ · experiment team make decisions Open-source tool from neutron scattering Plugin & interface for fusion data 9. Data Access: Simple Access Layer

24

Awarded RSE Fellowship

Promotion quickly followed

Can now dedicate herself fully to RSE work

▸ TexGen

▸ Disseminating software skills

Page 25: SOFTWARE ENGINEER THE RESEARCH€¦ · experiment team make decisions Open-source tool from neutron scattering Plugin & interface for fusion data 9. Data Access: Simple Access Layer

The story of the UK

RSE movement

25

Page 26: SOFTWARE ENGINEER THE RESEARCH€¦ · experiment team make decisions Open-source tool from neutron scattering Plugin & interface for fusion data 9. Data Access: Simple Access Layer

UK RSE Association

A Brief History…

Simon Hettrick:A very brief history of Research Software Engineersbit.ly/RSETalk2016

A not-so-brief history of Research Software Engineersbit.ly/NotSoBriefRSE

26

Page 27: SOFTWARE ENGINEER THE RESEARCH€¦ · experiment team make decisions Open-source tool from neutron scattering Plugin & interface for fusion data 9. Data Access: Simple Access Layer

Background…

2010

SSI

27

SSI set up to improve research software

Enabled RSE movement to get off the ground:

▸ Networking▸ Events▸ Publicity▸ Money▸ Links to funders, ▸ Research, policy, evidence

Page 28: SOFTWARE ENGINEER THE RESEARCH€¦ · experiment team make decisions Open-source tool from neutron scattering Plugin & interface for fusion data 9. Data Access: Simple Access Layer

Origins: SSI Collaborations Workshop 2012

2010

Term coined

Mar2012

SSI

“Developing the profession of a scientific software engineer and the career track of software developers in academia.”

- James Hetherington

The term “Research Software Engineer” adopted

28

Page 29: SOFTWARE ENGINEER THE RESEARCH€¦ · experiment team make decisions Open-source tool from neutron scattering Plugin & interface for fusion data 9. Data Access: Simple Access Layer

2010

Term coined

FirstRSG

Mar2012

Late2012

SSI

2012: First Research Software Group

First RSE Group at University College London

New model of organising software expertise in a University

29

Page 30: SOFTWARE ENGINEER THE RESEARCH€¦ · experiment team make decisions Open-source tool from neutron scattering Plugin & interface for fusion data 9. Data Access: Simple Access Layer

2010

Term coined

FirstRSG

UK RSE

Mar2012

Late2012

Early2014

SSI

2014: RSE Association Founded

RSE Workshop Sep 2013

“Should we create a community to represent RSEs?”

160 members in first 3 months

30

Page 31: SOFTWARE ENGINEER THE RESEARCH€¦ · experiment team make decisions Open-source tool from neutron scattering Plugin & interface for fusion data 9. Data Access: Simple Access Layer

2014: Elections, word spreads

2010

Term coined

FirstRSG

UK RSE

Democratic

Mar2012

Late2012

Early2014

Late2014

SSI

31

Page 32: SOFTWARE ENGINEER THE RESEARCH€¦ · experiment team make decisions Open-source tool from neutron scattering Plugin & interface for fusion data 9. Data Access: Simple Access Layer

2015: RSE Fellowships

Funding body creates dedicated RSE Fellowships

Five years funding 211 applicants - 7 fellowships awarded(2nd round in 2017)

2010

Term coined

FirstRSG

UK RSE

Democratic RSE fellows

Mar2012

Late2012

Early2014

Late2014

June 2015

SSI

32

Page 33: SOFTWARE ENGINEER THE RESEARCH€¦ · experiment team make decisions Open-source tool from neutron scattering Plugin & interface for fusion data 9. Data Access: Simple Access Layer

2010

Term coined

FirstRSG

UK RSE

Democratic RSE fellows

Expansion

Mar2012

Late2012

Early2014

Late2014

June 2015

2015-16

SSI

2015-16: More groups + coordination!

RSE Leaders forum:Networking, help & moral support for people setting up or running RSE Groups

RSE network coordinator (Claire Wyatt)

33

Page 34: SOFTWARE ENGINEER THE RESEARCH€¦ · experiment team make decisions Open-source tool from neutron scattering Plugin & interface for fusion data 9. Data Access: Simple Access Layer

34

2016: First RSE Conference

2010

Term coined

FirstRSG

UK RSE

Democratic RSE fellows

Expansion FirstConference

Mar2012

Late2012

Early2014

Late2014

June 2015

2015-16

Sep 2016

SSI

Page 35: SOFTWARE ENGINEER THE RESEARCH€¦ · experiment team make decisions Open-source tool from neutron scattering Plugin & interface for fusion data 9. Data Access: Simple Access Layer

2010

Term coined

FirstRSG

UK RSE

Democratic RSE fellows

Expansion FirstConference

Current activities

Mar2012

Late2012

Early2014

Late2014

June 2015

2015-16

Sep 2016

SSI

2017-18

35

Current activities

(More on this later)

Page 36: SOFTWARE ENGINEER THE RESEARCH€¦ · experiment team make decisions Open-source tool from neutron scattering Plugin & interface for fusion data 9. Data Access: Simple Access Layer

What do RSEs do?

36

Page 37: SOFTWARE ENGINEER THE RESEARCH€¦ · experiment team make decisions Open-source tool from neutron scattering Plugin & interface for fusion data 9. Data Access: Simple Access Layer

What do RSEs do?

▸ Build quality software - as part of research projects

▸ Help researchers

▹ Navigate computing infrastructure

▹ Learn skills

▹ Establish good practices

37

Page 38: SOFTWARE ENGINEER THE RESEARCH€¦ · experiment team make decisions Open-source tool from neutron scattering Plugin & interface for fusion data 9. Data Access: Simple Access Layer

Better software...

▸ Well designed, understandable

▸ Tested

▸ Documented and maintainable

▸ Easier to use, reuse and extend

▸ Better reproducibility

38

Page 39: SOFTWARE ENGINEER THE RESEARCH€¦ · experiment team make decisions Open-source tool from neutron scattering Plugin & interface for fusion data 9. Data Access: Simple Access Layer

39

Example project: CHERAB

Spectroscopy simulation framework

Modelling of hot plasma diagnostic data

Page 40: SOFTWARE ENGINEER THE RESEARCH€¦ · experiment team make decisions Open-source tool from neutron scattering Plugin & interface for fusion data 9. Data Access: Simple Access Layer

40

CHERAB: RSE input

Dr Matthew Carr (Physicist)Dr Alex Meakins (RSE)

Preparing CHERAB for worldwide use

▸ Improving architecture▸ Speeding up▸ Open-source▸ Packaging

Page 41: SOFTWARE ENGINEER THE RESEARCH€¦ · experiment team make decisions Open-source tool from neutron scattering Plugin & interface for fusion data 9. Data Access: Simple Access Layer

41

CHERAB: Impact

Page 42: SOFTWARE ENGINEER THE RESEARCH€¦ · experiment team make decisions Open-source tool from neutron scattering Plugin & interface for fusion data 9. Data Access: Simple Access Layer

RSE groups

42

Page 43: SOFTWARE ENGINEER THE RESEARCH€¦ · experiment team make decisions Open-source tool from neutron scattering Plugin & interface for fusion data 9. Data Access: Simple Access Layer

Research Software Groups

43

Flexible access to software expertise

Pooling resources for small projects

Central source of advice, training, infrastructure

Stable careers for RSEs

Peer group

Can maintain wider range of skills

Focus for a wider network

Page 44: SOFTWARE ENGINEER THE RESEARCH€¦ · experiment team make decisions Open-source tool from neutron scattering Plugin & interface for fusion data 9. Data Access: Simple Access Layer

Research software groups in UK

44

Page 45: SOFTWARE ENGINEER THE RESEARCH€¦ · experiment team make decisions Open-source tool from neutron scattering Plugin & interface for fusion data 9. Data Access: Simple Access Layer

The support and recognition provided by a central RSE group allowed Tania to develop her skills and career

45

Career case study: Dr Tania AllardPhD in materials science

Encountered issues with poorly-written research software

Joined the RSE group at the University of Sheffield

Page 46: SOFTWARE ENGINEER THE RESEARCH€¦ · experiment team make decisions Open-source tool from neutron scattering Plugin & interface for fusion data 9. Data Access: Simple Access Layer

Help setting up an RSE Group

http://rse.ac.uk/resources/how-to-start-an-rse-group/

RSE leaders discussions

Slack channel for people running or trying to start RSE groups or networks

RSE group case studies

46

Page 47: SOFTWARE ENGINEER THE RESEARCH€¦ · experiment team make decisions Open-source tool from neutron scattering Plugin & interface for fusion data 9. Data Access: Simple Access Layer

Research software

communities

47

Page 48: SOFTWARE ENGINEER THE RESEARCH€¦ · experiment team make decisions Open-source tool from neutron scattering Plugin & interface for fusion data 9. Data Access: Simple Access Layer

Communities within organisations

Networking Knowledge sharing Self-help

Raising the profile & status of research software work

48

Page 49: SOFTWARE ENGINEER THE RESEARCH€¦ · experiment team make decisions Open-source tool from neutron scattering Plugin & interface for fusion data 9. Data Access: Simple Access Layer

Institutional communities & events

49

▸ Southampton ▸ UCL ▸ Oxford ▸ Cambridge ▸ RAL/STFC ▸ Nottingham ▸ Newcastle▸ ORNL

“Research software community”

“Research Programming Hub”

“Research Software Developer Network”

“Computing seminar days”

“Software Expo”

Page 50: SOFTWARE ENGINEER THE RESEARCH€¦ · experiment team make decisions Open-source tool from neutron scattering Plugin & interface for fusion data 9. Data Access: Simple Access Layer

National RSE associations

50

Page 51: SOFTWARE ENGINEER THE RESEARCH€¦ · experiment team make decisions Open-source tool from neutron scattering Plugin & interface for fusion data 9. Data Access: Simple Access Layer

UK RSE: current activities

▸ Continued growth

▸ Bigger conference!

▸ Formal organisation

▸ International links

51

www.rse.ac.uk @ResearchSoftEng

Page 52: SOFTWARE ENGINEER THE RESEARCH€¦ · experiment team make decisions Open-source tool from neutron scattering Plugin & interface for fusion data 9. Data Access: Simple Access Layer

52

UK RSE membership

Page 53: SOFTWARE ENGINEER THE RESEARCH€¦ · experiment team make decisions Open-source tool from neutron scattering Plugin & interface for fusion data 9. Data Access: Simple Access Layer

The Society of Research Software Engineering

Currently registering as a “Charitable Incorporated Organisation”

▸ Legal entity▸ Independent of any individual or University▸ Elected trustees▸ Manage funds

53

Page 54: SOFTWARE ENGINEER THE RESEARCH€¦ · experiment team make decisions Open-source tool from neutron scattering Plugin & interface for fusion data 9. Data Access: Simple Access Layer

RSE Conference

Two successful events in 2016 & 17

> 200 participants from 14 countries

Talks, workshops, tutorials

54

Page 55: SOFTWARE ENGINEER THE RESEARCH€¦ · experiment team make decisions Open-source tool from neutron scattering Plugin & interface for fusion data 9. Data Access: Simple Access Layer

RSE18

September 3rd & 4thUniversity of Birmingham, UK

http://rse.ac.uk/conf2018/

Registration opens June 5th

55

Page 56: SOFTWARE ENGINEER THE RESEARCH€¦ · experiment team make decisions Open-source tool from neutron scattering Plugin & interface for fusion data 9. Data Access: Simple Access Layer

More national RSE associations

Gemany: DE-RSE

www.de-rse.org

Netherlands: NL-RSE

nl-rse.org

56

Page 57: SOFTWARE ENGINEER THE RESEARCH€¦ · experiment team make decisions Open-source tool from neutron scattering Plugin & interface for fusion data 9. Data Access: Simple Access Layer

National RSE Associations

Nordic Countries: NORDIC-RSE

www.nordic-rse.org

Under construction:

▸ Australia & New Zealand

▸ More in discussion...

57

Page 58: SOFTWARE ENGINEER THE RESEARCH€¦ · experiment team make decisions Open-source tool from neutron scattering Plugin & interface for fusion data 9. Data Access: Simple Access Layer

58

International RSE

movement

Page 59: SOFTWARE ENGINEER THE RESEARCH€¦ · experiment team make decisions Open-source tool from neutron scattering Plugin & interface for fusion data 9. Data Access: Simple Access Layer

International RSE activity

Conferences

RSE17

▸ Int. workshop▸ WSSSPE co-located

Sessions / talks in several national & international conferences

Networks& groups

Name and concept being adopted, eg

▸ Princeton, ORNL, USA

▸ Radboud, Netherlands

RSE Surveys

Comparative data from:

▸ Germany▸ Netherlands▸ UK▸ USA▸ South Africa▸ Canada

59

Page 60: SOFTWARE ENGINEER THE RESEARCH€¦ · experiment team make decisions Open-source tool from neutron scattering Plugin & interface for fusion data 9. Data Access: Simple Access Layer

International RSE leaders workshop

30th-31st January 2018

Bringing together RSE leaders from around the world to help each other improve access to software expertise in research by pooling knowledge, coordinating efforts and establishing collaboration.

60

Page 61: SOFTWARE ENGINEER THE RESEARCH€¦ · experiment team make decisions Open-source tool from neutron scattering Plugin & interface for fusion data 9. Data Access: Simple Access Layer

Participants Running/starting RSE groups & networks

Leading research software initiatives

61

Page 62: SOFTWARE ENGINEER THE RESEARCH€¦ · experiment team make decisions Open-source tool from neutron scattering Plugin & interface for fusion data 9. Data Access: Simple Access Layer

Coordinating efforts, sharing experience

Sharing group models RSE evidence bank

62

Page 63: SOFTWARE ENGINEER THE RESEARCH€¦ · experiment team make decisions Open-source tool from neutron scattering Plugin & interface for fusion data 9. Data Access: Simple Access Layer

researchsoftware.org

International RSE websiteBlog post published

Blog posts:

▸ Workshop

▸ How to set up a national RSE Association

▸ Tools for managing RSE projects

63

Page 64: SOFTWARE ENGINEER THE RESEARCH€¦ · experiment team make decisions Open-source tool from neutron scattering Plugin & interface for fusion data 9. Data Access: Simple Access Layer

64

Getting involved

Page 65: SOFTWARE ENGINEER THE RESEARCH€¦ · experiment team make decisions Open-source tool from neutron scattering Plugin & interface for fusion data 9. Data Access: Simple Access Layer

Request invitation via www.rse.ac.uk

(Or ask Scott Henwood / UK RSE people in the break!)

65

Page 66: SOFTWARE ENGINEER THE RESEARCH€¦ · experiment team make decisions Open-source tool from neutron scattering Plugin & interface for fusion data 9. Data Access: Simple Access Layer

Canadian Research Software Development

CRSC18

May2018

RSE Survey

2017 2019

What’s next?

How can you help?

?

66

Page 67: SOFTWARE ENGINEER THE RESEARCH€¦ · experiment team make decisions Open-source tool from neutron scattering Plugin & interface for fusion data 9. Data Access: Simple Access Layer

PICK WHAT FITSAction can be taken on all scales

67

You know your context best

Page 68: SOFTWARE ENGINEER THE RESEARCH€¦ · experiment team make decisions Open-source tool from neutron scattering Plugin & interface for fusion data 9. Data Access: Simple Access Layer

CHANGE ISHARD

68

Page 69: SOFTWARE ENGINEER THE RESEARCH€¦ · experiment team make decisions Open-source tool from neutron scattering Plugin & interface for fusion data 9. Data Access: Simple Access Layer

COMMUNITIES ARE POWERFUL

69

Page 70: SOFTWARE ENGINEER THE RESEARCH€¦ · experiment team make decisions Open-source tool from neutron scattering Plugin & interface for fusion data 9. Data Access: Simple Access Layer

BIG GOAL SMALL STEPSKeep going and small changes add up

70

Page 71: SOFTWARE ENGINEER THE RESEARCH€¦ · experiment team make decisions Open-source tool from neutron scattering Plugin & interface for fusion data 9. Data Access: Simple Access Layer

MAKE IT HAPPEN!

71

Page 72: SOFTWARE ENGINEER THE RESEARCH€¦ · experiment team make decisions Open-source tool from neutron scattering Plugin & interface for fusion data 9. Data Access: Simple Access Layer

Simon Hettrick:A very brief history of Research Software Engineersbit.ly/RSETalk2016

A not-so-brief history of Research Software Engineersbit.ly/NotSoBriefRSE

[email protected]@alysbrett

[email protected]@ResearchSoftEng

Read more www.rse.ac.uk

www.researchsoftware.org

RSE State of the Nation Reportdoi.org/10.5281/zenodo.495360

72

Presentation template by SlidesCarnivalModified for UK RSE use by Mark Turner