Embed Size (px)
Citation preview
D2.2 Customer Feedback Measurement Version 0.8
Document Information
Contract Number 676553
Project Website www.pop-coe.eu
Contractual Deadline M12, October 2016
Dissemination Level PU
Nature R
Author D. Loureiro (INRIA)
Contributor(s) JM Morel (Teratec)
Reviewer B. Mohr (JUELICH)
Keywords Customer Advocacy, Customer Feedback, User Forum
Notices: The research leading to these results has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No “676553”.
2015 POP Consortium Partners. All rights reserved.
D2.2 Customer Feedback Measurement Version 0.8
2
Change Log Version Author Description of Change
V0.1 D. Loureiro Initial Draft
V0.2 B. Mohr 1st review
V0.3 JM Morel
Update taking into account Bernd’s remarks and change in deliverable template.
Highlight of main lessons learnt via the customer satisfaction survey
V0.4 D. Loureiro Update of the sections 2.2, 2.3 and 4
V0.5 JM Morel Minor updates
V0.6 B. Mohr 2nd review
V0.7 Maria Toth Complementary review
V0.8 JM Morel Update taking Maria’s remarks into account
D2.2 Customer Feedback Measurement Version 0.8
3
Table of Contents
Executive Summary..................................................................................................................4
1. Introduction ......................................................................................................................4
2. Customer Survey, setup and feedback .........................................................................5
2.1 Setup of the customer feedback through surveys .........................................................5 2.2 Customer feedback .......................................................................................................5 2.3 Interviews .......................................................................................................................7
3. User Forum Meeting: BoF session at ISC’16 ................................................................8
4. First suggestions to POP management .........................................................................9
5. Conclusions................................................................................................................... 10
Acronyms and Abbreviations ............................................................................................... 11
List of Figures ........................................................................................................................ 12
D2.2 Customer Feedback Measurement Version 0.8
4
Executive Summary
This deliverable summarizes the findings of the work package “Customer Advocate” during the first year of the project. It includes a section reporting on the first User Forum Meeting, and presents the actual suggestions made by the Customer Advocate to the internal Operational Management meetings of the projects.
1. Introduction
As stated in the proposal “The Customer Advocacy is a way of ensuring that the activities of the project are really performed to the full satisfaction of the customers. POP will deliver value to application developers, infrastructure operators and scientific users of parallel applications. […] The Customer Advocacy will gather their feedback and make sure they play an important role in driving the operation of the Centre of Excellence. To this end, The Customer Advocacy will carry out a process for measuring Customer Satisfaction and will organise events where customers can freely express their feedback and suggestions.” To this end, several actions have been conducted:
The selection of SurveyMonkey to get the feedback of end-users via dedicated questionnaires which have been designed by the work package Customer Advocacy (WP2) people and reviewed by all POP members
The setup of various surveys in order to get the feedback of the end-users for each specific action (Audit of code, Performance Plan, Proof-of-Concept) and to get feedback about the Performance analysis tools and on the ROI gained by the user when the code has been optimized.
The creation of e-mail templates in order to standardize the interaction with the users
The systematic mailing of a synthetic document to end-users gathering their answers to the survey and the feedback from the interviews
The organization of a BoF session at ISC 2016 to present the project and give some feedback on the first performance assessments performed and the corresponding ROI for the users
These set of actions have already allowed the Customer Advocacy to provide to the project a synthesis of the first customer feedbacks in the first year of the project. Finally, scripts for interviews have been written in order to get more in-depth customer feedbacks on some specific aspects of the audits, performance plans and Proof of Concepts.
D2.2 Customer Feedback Measurement Version 0.8
5
2. Customer Survey, setup and feedback
2.1 Setup of the customer feedback through surveys
As described in the D2.1 Deliverable, the Customer Advocacy finally decided to use SurveyMonkey1. SurveyMonkey is a web-based service allowing the creation of surveys in a pretty efficient way and providing an interesting administration interface with graphs and exportations capabilities in various formats for external statistics or publication. In order to get the customer feedback efficiently, the following surveys have been designed:
Three questionnaires related to POP services (one for each type of service):
Performance Audit: https://fr.surveymonkey.com/r/F1_Report
Performance Plan: https://fr.surveymonkey.com/r/F2_Plan
Proof Of Concept: https://fr.surveymonkey.com/r/F3_PoC
One questionnaire regarding Performance tools (for users who are using the performance analysis tools by themselves to measure the performance improvement resulting from code modifications) https://fr.surveymonkey.com/r/F4_Perf-Tools
One regarding the evaluation of the performance improvement and of the resulting gains in order to measure the global return on investment https://fr.surveymonkey.com/r/F5_Perf-gains
Note: While the draft version of these questionnaires have been described in the Deliverable D2.1 (delivered at T0+3), the final versions can be found by clicking on the above urls.
2.2 Customer feedback
As part of the process providing POP services to end-users, when an audit, a performance plan or a proof of concept is finished, the corresponding survey is sent to the end-user. After the survey is filled in, the compilation of the answers is sent to the end-user who filled it in. Various e-mail templates have been created to send the surveys, to follow up when the survey has not been filled in, and to thank the end-user when it is done.
1 https://fr.surveymonkey.net/?ut_source=header
D2.2 Customer Feedback Measurement Version 0.8
6
So far, there is a pretty good return rate of the completed surveys (~78%, 18/23), partly because the questionnaires regarding the audit, the performance plan, and the proof-of-concept are quite straightforward to complete and because nonresponding end-users are rapidly called back. As of today (Sept. 7, 2016), 18 answers have been gathered: 15 regarding audit reports, 1 for Performance Plan and 2 for Proof-of-Concept. The respective data summaries for the Analysis Report, the Performance Plan and the Proof-of-Concept can be found in the annex 1, 2 and 3 respectively. If we take a look at the answers related to the general perception of POP experts and their reports for the Audit service, the feedback is very good:
The end users found that the POP experts were very to extremely responsive when questioned about the Audit (see Figure 1)
The customers found that the answers of the POP experts to their questions were from good to excellent (see Figure 2)
Figure 1 How responsive have the POP experts been to your questions or concerns about the analysis and the report?
Figure 2 What was the quality of their answers?
Even if we received just 3 answers for the Performance Plan and Proof-Of-Concept services so far, the corresponding levels of satisfaction are the same. The feedback also includes interesting remarks like the following ones:
“The Report is well structured and showed me different points where to optimize my code. Some of them were clear to me but others were quite surprising, but helped me a lot to further improve the performance of our application”.
This clearly shows that while the user was aware of some problems, the audit enabled him to discover a few others he did not suspect.
The POP audits help to identify problems in many aspects:
o Inefficient use of simultaneous threads
o Lack of (or inefficient) use of vectorization
o Inefficient use of memory
o Inefficient communications (MPI, I/O, ...)
D2.2 Customer Feedback Measurement Version 0.8
7
o Redundant operations
o Inefficient allocation strategy
o Load imbalances
o …
More than one user out of three would like to get training, especially in performance analysis.
60% of the users who got an Audit would like to continue with a Performance Plan (see Figure 3). This is good news, in particular because the first audit reports included some recommendations to fix the problems identified that possibly made the user believe that a more detailed Performance Plan was not needed.
Figure 3 Are you going to proceed with a next step (ask for a Performance Plan and/or a Proof-of-Concept)?
Note: Of course, POP experts will consider the requests from those who ask for a
Performance Plan and/or a Proof of Concept, but we will also go back, a few months
later, to those who think they got enough information, to check whether they could
implement the recommendation or if they need some help.
The evaluations of the users who got a Performance Plan and/or a Proof-of-Concept are quite encouraging: they are very satisfied and declare that they would be ready or possibly ready to pay.
If we take a closer look at the type of structure submitting codes for the Audit, only 4 out of 23 are companies. The remaining part is composed of universities and laboratories.
2.3 Interviews
In order to get a deeper understanding of the customer experience, several scripts for interviews have been written. The first interviews are currently being realized and appear useful to complete the feedback gathered via questionnaires.
Yes a Performance Plan Yes a Proof of Concept No I have got enough
D2.2 Customer Feedback Measurement Version 0.8
8
At least one customer of each POP Experts Centre (BSC, JSC, HLRS, RWTH, and NAG) having achieved Performance Audit services will be interviewed. The same will apply regarding Performance Plan and Proof-of-Concept services.
3. User Forum Meeting: BoF session at ISC’16
A BoF session was organized by POP during the last ISC High Performance 2016 conference held in Frankfurt, on Wednesday, June 22, 2016. This BoF session was mainly dedicated to:
Code developers interested in the assessment of detailed actual behaviour of their code and to learn about suggestions of most productive directions to refactor the code.
HPC code users interested in the assessment of achieved performance in specific production conditions or to learn about possible improvements by modifying their environment setup.
Infrastructure operators interested in the assessment of achieved performance of codes running in production conditions, training of their support staff, or getting feedback for time computer time allocation processes.
Vendors interested in benchmarking, customer support and system dimensioning/design.
Figure 4 Jesus Labarta introducing the POP CoE to the audience during the BoF
session at ISC'16
First, POP coordinator Jesus Labarta introduced the POP CoE to the audience in a short overview. Next, Mike Dewar, work package leader "POP Community Development", reported on first results from the performance
D2.2 Customer Feedback Measurement Version 0.8
9
audit, performance plan, and proof-of-concept services provided by POP in the first nine months of the project. The BoF concluded with two presentations by two satisfied POP customers: Giovanni Erbacci from Cineca presented results from the performance audit of the QuantumEspresso code. Finally, Xavier Vigouroux from Atos reported on the outcome of the performance audit of the NEMO code. The customers (Cineca and Atos) were very supportive and as it can be seen in their presentations, they were very positive about the POP service they received and invited the audience to also try it.
4. First suggestions to POP management
If we take a deeper look at the answers provided by the end-users some points could be improved:
Even if most of the end-users explain that the reports where clear and sound, there is still two person stating that the report was, in a first approach, somewhat difficult to understand. The provided comments give some details, but more legends, units or explanations would have allowed the end-users to get a perfect understanding. However, discussions with the experts solved their issues rapidly and the corrections allowed them to get a clear understanding of the reports.
More explanations in the reports could be an improvement.
A customer reported in a comment that “Analysis should be made for a larger number of threads / MPI cores. It is not clear whether conclusions drawn from a very small number of cores can be extended to real-world simulations with thousands of cores.”
Make user aware of the fact that the performance analysis must be made on a configuration which is similar to the configuration that is used for exploitation. Note: This is already partly solved by the updated service questionnaire for the customer which now asks for a detailed description of both production runs and development test case configurations.
Another customer comment showed that “More frequent and early exchange on arising questions regarding obstacles found in the code. Some issues could have been resolved easily by an earlier communication.”
A closer interaction with the end-user could be a good improvement to the services.
A small number of codes that have been audited where then subject of Performance Plan or Proof-of-Concept despite the fact that 60% of the customers would like to go forward with a Performance Plan and/or a Proof-of-Concept.
D2.2 Customer Feedback Measurement Version 0.8
10
Since the Performance Plan is considered as a key to understand how to optimize the code, from now on POP experts must strongly recommend a Performance Plan as a follow-up of the Audit each time they detect a room for performance improvement.
The same applies for Proof-of-Concept whose potential value has been so far a bit underestimated by users.
Only 4 out of 23 customers that answered the survey were companies.
An effort must be made to augment the ratio of companies/ISVs/SME in the pool of customers.
5. Conclusions
During the first twelve months several audit reports, performance plans and proof-of-concepts have been performed. Following these services, the Customer Advocate did several follow-ups with the users in order to get their feedback. This feedback was very good and showed that the users were satisfied with the services provided by POP and several of them even choose to go forward with POP through a Performance Plan or a Proof-of-Concept Service after an Audit was performed. The User Meeting Forum organized as a BoF session during the last ISC High Performance conference was also very useful to create some awareness on the project, to provide information to potential customers about the first studies performed and get some feedback from end-users that were present during the session. Finally, some improvements could be identified from the first audits and the feedback gathered through the surveys. Those improvements will certainly help the project to be even more effective in the various services provided.
D2.2 Customer Feedback Measurement Version 0.8
11
Acronyms and Abbreviations
- AR – Audit Report - BSC – Barcelona Supercomputing Center - CA – Consortium Agreement - CAdv – Customer Advocate - DoA – Description of Action (Annex 1 of the Grant Agreement) - EC – European Commission - GA – General Assembly / Grant Agreement - HLRS – High Performance Computing Centre (University of Stuttgart) - HPC – High Performance Computing - IPR – Intellectual Property Right - Juelich – Forschungszentrum Juelich GmbH - KPI – Key Performance Indicator - MS – Milestones - PEB – Project Executive Board - PM – Person month / Project manager - PoC – Proof-of-Concept - POP – Performance Optimization and Productivity - PP – Performance Plan - RV – Review - RWTH Aachen – Rheinisch-Westfaelische Technische Hochschule
Aachen - USTUTT (HLRS) – University of Stuttgart - WP – Work Package - WPL – Work Package Leader
D2.2 Customer Feedback Measurement Version 0.8
12
List of Figures
Figure 1 How responsive have the POP experts been to your questions or concerns about the analysis and the report? ................................................... 6
Figure 2 What was the quality of their answers? ............................................. 6
Figure 3 Are you going to proceed with a next step (ask for a Performance Plan and/or a Proof-of-Concept)? .................................................................... 7
Figure 4 Jesus Labarta introducing the POP CoE to the audience during the BoF session at ISC'16 ...................................................................................... 8
