Embed Size (px)
Citation preview
Systematic determination of a merit, worth and significance of productCriteria based on a set of standardsDegree of achievement
objectives v.s. resultsTailored to its context
Assessment of a product quality
Activities CAPTURE – collecting dataANALYSIS – interpreting data to identify problemsCRITIQUE – suggesting solutions or improvements
to mitigate problems2
A language is a means of communication
The user interface is a realization of a languageA language is a model that describes the
allowed terms and how to compose them into valid sentences 3
General Purpose (programming) Languages (GPLs)User has to master programming concepts User has to master domain concepts
Domain Specific (modeling) Languages (DSLs)Meant to close gap between PROBLEM DOMAIN and
SOLUTION DOMAINReduce the use of computation domain conceptsFocus on the domain concepts
4
VerificationDid I build the thing right?Is the right product functionality provided?
(from language engineer understanding)Focus is on the language
ValidationDid I build the right thing?Is end user satisfied with product?Focus is NOT on the language’s users
Should this be the other way around?5
Increasingly popularRaise the abstraction level (closer to the domain)Narrow the design space
Several benefits claimed, in well-defined domainsProductivity gainsBetter time to marketAvoid error-prone mappings between domain and
software development concepts Leverage the expertise of domain experts
6
[Mernik, 2005]7
REUSE?
The capability of a software product to enable specified users to achieve specified goals with: effectiveness, productivity, safety and satisfaction in specified contexts of use
8[ISO IEC 25010]
Dynamic, structured information space that includes the following entitiesa model of the User
Different knowledge sets Characteristics chosen are dependent on application
domainthe hardware-software Platform
set of computing, sensing, communication, and interaction resources
e.g. operating systems, memory size, network bandwidth, input and output interaction devices
the social and physical Environment Where the interaction is actually taking place
Different languages may have different contexts of useTheir users are likely to have different knowledge
setsA minimum set of ontological concepts is required
to use the language
9
The user's view of the Quality of a productMeasured in terms of the result of using
the product, rather than its properties
10
Formal evaluation Models and simulations to predict measures of usability Some can be used before a prototype is available
Automatic evaluation Automated conformance checking to guidelines and standards Requires at least a prototype, or an initial version of the full
implementationEmpirical evaluation
Possible at any development stage Requires users Formative methods (e.g. think aloud) vs. Summative methods
(using metrics)Heuristic evaluation
Evaluation conducted by experts (often before userrs are involved)
Without scenarios: reviews, inspections With scenarios (task based): walkthroughs
11
MUSiC – Metrics for Usability Standards in ComputingUser satisfaction method (questionnaire-based)Performance measurement method (observation-based)Cognitive workload measurement method (questionnaire-
based)Analytic measurement method (dynamic model analysis +
simulation tools)MAGICA
User satisfaction measurement (questionnaire-based)Task completion time measurement (video)Cognitive effort (questionnaire-based)Heuristic adherence evaluation (analysis)
UCA – Usability Context AnalysisContext report form (stakeholder meeting)Context of evaluation – products users (stakeholder meeting)Context analysis (stakeholders meetings) 12
Identify need for
user-centred design
Understand and specify
the context of use
Evaluate designs against
requirements
Produce design
solutions
Specify the user and
organizational requirements
System meets specified
functional, user and
organizational requirements
13
• To evaluate, or not to evaluate • (aka “Should we?”)
• Facts are facts, even when portrayed by statistics
• (aka “Do we?”)• How do language engineers
evaluate languages?• (aka “How can we?”)
• Language evaluation forensics • (aka “Life in the trenches”)
14
Barišić, Amaral, Goulão, and Barroca: ‘Evaluating the Usability of Domain-Specific Languages’, (IGI Global, 2012)
DSL development is hard Requires domain and language development expertise
Many DSL development techniques which should we use?
CostfullNo systematic aproachNo awarness of Software Language Engineering process
ChallengesDevelopment of training materialsSupportStandardizationMaintenance
15
Evaluating candidate DSL• Building/adopting DSL• Developing evaluation and
training materials• Training/Evaluation• Establishing a baseline for
comparing performance with the DSL
Not evaluating candidate DSL• Inability to estimate return on
investment in the adoption of the DSL
• What is the break even point?• What is the DSL’s impact on
the process quality?• What is the DSL´s impact on
the product quality?
16
Simply NOT true…e.g. Language Level has been around, and widely used,
since 1996
Language evaluation has been a concern for many decades. For instance,
“…the tools we are trying to use and the language or notation we are using to express or record our thoughts are the major factors determining what we can think or express at all! The analysis of the influence that programming languages have on the thinking habits of their users … give[s] us a new collection of yardsticks for comparing the relative merits of various programming languages.”
[Dijkstra 1972]17
18
Is Perl better than Python?
Code! Yes. A programmer's strength flows
from code maintainability.
But beware of Perl.
Terse syntax... more than one way to do it...
default variables.
The dark side of code
maintainability are they.
Easily they flow, quick to join you when code you
write.
If once you start down the dark path, forever
will it dominate your destiny,
consume you it will.
No... no... no. Quicker, easier, more seductive.
But how will I know why Python is better
than Perl?You will know. When your code you try to read
six months from now.
http://www.netfunny.com/rhf/jokes/99/Nov/perl.html
Language QualitiesClarity, simplicity, and unity of language conceptClarity of program syntaxNaturalness for the applicationSupport for data abstractionEase of program verificationProgramming environmentPortability of programsCost of program executionCost of program translationCost of program creation, testing, and useCost of program maintenance
19
[Pratt 1984]
Language and its documentation qualities Completeness of definition Independence from hardware Modularization and support for abstraction Smallness of size Conciseness and clarity of description
Implementation qualities Reliability Compilation speed Efficiency of code Predictability of execution cost Compactness of compiled code Simple and effective interface to environment
20
[Wirth 1984]
Language design and implementation criteriaIs the language formally defined?Is the language unambiguous?
Human factors criteriaDo programmers easily write correct, understandable
code in the language?How easy is the language to learn?
Software Engineering criteriaSupport for quality attributes such as portability,
reliability, maintainability...Availability of good tools and experienced
programmersApplication domain criteria
How well does the language support programming for specific applications?
21
[Howatt 1995]
Project-specific criteriaEven within a domain, specific projects
will have specific requirements Criteria should be defined within projects Criteria should have an evaluation richer than just
yes/no, e.g. (criterium, satisfaction score, importance score)
RelevanceExternal constraints are also relevant,
e.g. Legacy code Use what everybody else is using (should be good,
right?) Language availability Contractual obligations
22
[Howatt 1995]
In general, software language engineers do not evaluate their languages with respect to their impact in the software development process in which the DSLs will be integratedOr, if they do, they are extremely shy about it…
23
[Gabriel 2010]
Is there a concrete and detailed evaluation model to measure DSLs Usability?
Is the DSL community concerned about experimental evaluation as a mechanism to prevent future problems emerging from the proposed DSLs?
To what extent does the DSL community present evidence that the developed DSLs are easy to use and correspond to end-users needs?
24
[Gabriel 2010]
RQ1: Does the paper report the development of a DSL?
RQ2: Does the paper report the DSL development process with some detail?
RQ3: Does the paper report any experimentation conducted for the assessment of the DSL?
RQ4: Does the paper report the inclusion of end-users in the assessment of a DSL?
RQ5: Does the paper report any sort of usability evaluation?
25
[Gabriel 2010]
26
Selection Publication
Available
Inspected articles
Selected articles
Selection Percentage
Direct OOPSLA-DSM
97 97 14 14.4%
OOPSLA-DSVL
5 27 5 18.5%
DSPD 19 19 3 15.8%SLE 18 18 0 0.0%ATEM 13 13 2 15.4%MDD-TIF 10 10 3 30.0%DSML 12 10 0 0.0%OOPSLA-SF 9 9 0 0.0%ECOOP-ERLS
6 6 0 0.0%
JVLC 5 5 2 40.0%
Query-based search*
VL/HCC 141 16 2 12.5%
LDTA 10 2 1 50.0%MODELS 200 4 1 25.0%ICSE 42 6 2 33.3%TSE 32 2 1 50.0%
Total 641 242 36 14.6%2001-2008 [Gabriel 2010]
Few papers (14%) report any sort of evaluation Even those provide too
few details Too much tacit
knowledge: virtually impossible to replicate evaluations and perform meta-analysis
Predominance of toy examples Unsubstantiated claims to
the merits of DSLsPoor characterization of
subjects involved in validation How representative are
they of real DSL users? 27
28
What to measure?
29
30[Barisic, 2011a]
Requirements definition
Design plannin
gData
collection
Data analysis
Result packaging
31[Barisic, 2012]
32
Suggestive perception of the quality characteristics set that will influence Usability of DSLs
[Barisic, 2011a]
33
[Barisic, 2011c]
34[Barisic, 2011a]
• Introduce DSLs’ Usability evaluation during DSLs’ life-cycle iterations • Design an effective experimental evaluation of DSLs that will provide
qualitative and quantitative feedback for DSLs developers• Produce user-centered design of DSL• Foresee the Quality of a DSL while in an iterative evolution step• Merge the Software Language development process with the Usability
Engineering process
35
Barišić, Monteiro, Amaral, Goulão, Monteiro: "Patterns for Evaluating Usability of Domain-Specific Languages“, InProceedings of the 19th Conference on pattern languages of programs (PLoP), SPLASH 2012 Tucson, Arizona, USA, October 2012
36
Barišić, Monteiro, Amaral, Goulão, Monteiro: "Patterns for Evaluating Usability of Domain-Specific Languages“, InProceedings of the 19th Conference on pattern languages of programs (PLoP), SPLASH 2012 Tucson, Arizona, USA, October 2012
37
Barišić, Monteiro, Amaral, Goulão, Monteiro: "Patterns for Evaluating Usability of Domain-Specific Languages“, InProceedings of the 19th Conference on pattern languages of programs (PLoP), SPLASH 2012 Tucson, Arizona, USA, October 2012
38
Language
engineer
Domain
expert
Usability
engineer
39
40[Barisic 2012]
Evaluation sessionper language per group
41 41[Barisic2011b]
Two types of physicists (graduated students) involvedInformed programmers (Inf) – regular users of
programming languages and they are used to program with the present analysis framework
Uninformed programmers (non-Inf) - regular users of programming languages and they are not used to program with the present analysis framework
42[Barisic2011b]
43
Features we wanted to have evaluated:query steps in Pheasant vs. C++/BEE expressing a decay specification of filtering conditions vertexing and the usage of user-defined
functions aggregation path expression (navigation queries) expressing the result set the expressiveness of user-defined
functions[Barisic2011b]
Our evaluation technique was tested with two individuals (two physics experts) in order to verify it and to test the teaching materials and questionnaires
As time constrants and equipment turn out to be adequat there was no need to change prepared materials
44[Barisic2011b]
RQ1:Is querying with Pheasant more effective than with C++/BEE?
RQ2:Is querying with Pheasant more efficient than with C++/BEE?
RQ3:Are participants querying with Pheasant more confident on their performance than with C++/BEE?
Our goal is to:analyze the performance of Pheasant programmers plug-
ins for the purpose of comparing it with a baseline
alternative (C++/BEE)with respect to the efficiency, effectiveness and
confidence of defying queries in Pheasant from the point of view of a researcher trying to assess
the Pheasant DSL, in the context of a case study on selected queries
45[Barisic2011b]
H1null Using Pheasant vs. C++/BEE has no impact on the effectiveness of querying the analysis framework
H1alt Using Pheasant vs. C++/BEE has a significant impact on the effectiveness of querying the analysis framework
H2null Using Pheasant vs. C++/BEE has no impact on the efficiency of querying the analysis framework
H2alt Using Pheasant vs. C++/BEE has a significant impact on the efficiency of querying the analysis framework
H3null Using Pheasant vs. C++/BEE has no impact on the confidence of querying the analysis framework
H3alt Using Pheasant vs. C++/BEE has a significant impact on the confidence of querying the analysis framework
46[Barisic2011b]
We focus on presenting six examples, each focusing in some of the features we chose to evaluate
Participants are asked to give themselves a mark for feeling of correctness of their trial
Session take the time needed for each group to understand the examples
47[Barisic2011b]
Every participant has four queries, specified in English, to be rewritten in previously learned language
Subjects makes self-assessment of his replay rating his feeling of correctness
Example:Build the decay of a D0 particle to a Kaon Pion
48[Barisic2011b]
49
Query solution in C++/BEE
(pseudo code based on real code)
Query solution in Pheasant
The participants were asked to judge the intuitiveness, suitability and effectiveness of the query language. The goal was to evaluate:Overall reactions Query language constructs
Affect to query language was rated by:Query language constructsParticipants’ comments
50[Barisic2011b]
Results obtained with Pheasant were clearly better then those with C++/BEE
Pheasant allowed non-programmers to correctly define their queries.
The evaluation also showed a considerable speedup in the query definition by all the groups of users that were using Pheasant
The feed-back obtained from the users was that it is more comfortable to use Pheasant than with the alternative.
51[Barisic2011b]
Results obtained with Pheasant were clearly better then those with C++/BEE
Pheasant allowed non-programmers to correctly define their queries.
The evaluation also showed a considerable speedup in the query definition by all the groups of users that were using Pheasant
The feed-back obtained from the users was that it is more comfortable to use Pheasant than with the alternative.
52[Barisic2011b]
EFFECT
IVEN
ESS
Erro
r r
ates
Stat
isti
cal
mea
ning
fuln
ess
53[Barisic2011b]
EFFICIEN
CYTi
me
Stat
isti
cal
mea
ning
fuln
ess
54[Barisic2011b]
CONFIDE
NCE
Self
ass
essm
ent
(0-5
)St
atis
tica
l m
eani
ngfu
lnes
s
55
Mean confidence / queryNon -
InfC++/BEE 1,04Pheasant 4,75
InfC++/BEE 4,88Pheasant 4,83
[Barisic2011b]
56
57
Literature [Mernik2005] M. Mernik, J. Heering, and A. M. Sloane: When and how to
develop domain-specific languages, 2005, ACM Computing Surveys [Gabriel2010] Gabriel, P., Goulão, M. & Amaral, V. (2010). Do Software
Languages Engineers Evaluate their Languages? in XIII Congreso Iberoamericano en "Software Engineering" (CIbSE'2010)
[Barisic2011a] Barišić, A., Amaral, V., Goulão, M., and Barroca, B.: ‘Quality in Use of DSLs: Current Evaluation Methods’. Proc. 3rd INForum - Simpósio de Informática (INForum2011), Coimbra, Portugal, September 2011
[Barisic2011b] Barišić, A., Amaral, V., Goulão, M., and Barroca, B.: ‘Quality in Use of Domain Specific Languages: a Case Study’. Proc. Evaluation and Usability of Programming Languages and Tools (PLATEAU) Portland, USA, October 2011
[Barisic2011c] Barišić, A., Amaral, V., Goulão, M., and Barroca, B.: ‘How to reach a usable DSL? Moving toward a Systematic Evaluation’, Electronic Communications of the EASST, 2011
[Barisic2012] Barišić, A., Amaral, V., Goulão, M., and Barroca, B.: ‘Evaluating the Usability of Domain-Specific Languages’, in Mernik, M. (Ed.): ‘Formal and Practical Aspects of Domain-Specific Languages: Recent Developments’ (IGI Global, 2012)
[Barisic2013] Barišić, A: ‘Evaluating the Usability of Domain-Specific Languages’, in Mernik, M. (Ed.): ‘Formal and Practical Aspects of Domain-Specific Languages: Recent Developments’ (IGI Global, 2012) 58
