Semi-Automated Design Guidance Enhancer (SADGE) A Framework for Architectural Guidance Development Mohsen Anvaari Norwegian University of Science and Technology

Semi-Automated Design Guidance Enhancer (SADGE)

A Framework for Architectural Guidance Development

Mohsen AnvaariNorwegian University of Science and Technology

Trondheim, [email protected]

Olaf ZimmermannUniversity of Applied Sciences of Eastern Switzerland

Rapperswil, [email protected]

Agenda

• Introduction• An Example Scenario• SADGE Framework Development• SADGE Framework Operation and Maintenance• Framework Evaluation – Setup• Framework Evaluation – Results• Discussion• Related Work• Conclusions• Future Work• Questions


Introduction• Architectural decision making is a non-trivial task for architects • Researchers have developed several concepts, methods and tools to

assist practitioners• One of the promising approaches is a decision identification

technique • It enhances architectural guidance (decisions required) from

decisions made previously• Practitioners still are reluctant to apply this technique • SADGE automatically extracts architectural issues from architectural

related documents by applying natural language processing first • Then, it manually post processes and fine-tunes the extracted

knowledge entities


An Example Scenario

• The book is a 500 pages text, and architect “A” has spent 50 hours to manually annotate 200 sentences that contain architectural issues

• SADGE assists the architects to automatically annotate the sentences that contain architectural issues

SADGE Framework Development

Automated Annotator: We used ANNIE and JAPE from an open source tool called GATE (general architecture for text engineering)


Annotating Rules and Catalog of Terms

High Priority Termsagree on, choose

Low Priority Terms

approach, articulate, class, component, construct, concern, define, designdetermine, different, employ, establish, evaluate, exchange, facilitateframework, investigate, limitation, make, philosophy, principle, profile

provide, protocol, recommend, refactor, require, schema, select, serviceseveral, strategy, support, topology, transaction management, type, various


SADGE Framework Operation and Maintenance

Framework Evaluation

• Purpose: Evaluate SADGE by comparing the three quality attributes (Precision, Recall, Effort) of manual, automatic and semi-automatic approaches of decision identification – Recall: A∩B/B– Precision: A∩B/A

Framework Evaluation• Participants: 19 students of a bachelor’s program in

information technology– randomly selected, familiar with the software architecture, but not

familiar with the concept of architectural knowledge

• We divided them into two groups of ten and nine students • Stages: In the first stage, students were supposed to annotate

the sentences they think are architectural issues • In the second stage, they were asked to reject the sentences

they disagreed with automated annotator to fine-tune the results


Framework Evaluation - Material• The document we selected for

the experiment was the one that had the smallest deviation from the mean of precision and recall of annotating all of the tested documents and therefore can be considered as a representative of the tested documents

• The right figure shows how the output of the experiment can look like


Framework Evaluation - Results


Discussion• The effort reduction results are in correspondence to our

expectation• The recall and precision of student annotations are very low (38 and

25 respectively) • In the industrial projects we expect that both precision and recall

should be higher• The results for automated annotator are relatively high (86 and 57) • These show that if the people in charge of enhancing architectural

guidance are not expert enough, automated approach will perform more accurate and more sensitive by spending much less time

• The other part that is not the same as our expectation is the recall rate of semi-automatic approach. The reason may lie in the expertise level of the participants


Related Work• Using NLP for knowledge extraction is not novel in software engineering

– Existing tools and methods for generating object oriented models from natural language texts by applying NLP (Perez-Gonzalez, 2002)(Harmain and Gaizauskas, 2003)(Bajwa et al., 2009)(Soeken et al., 2012)

• The majority of work has been done to extract the object oriented data from the documents

• Extraction of architectural knowledge (specifically architectural decisions) is not mainly in focus

• (Figueiredo et al., 2012) approach and TREx (López et al., 2012) are two works applying NLP in AK

• The catalog of terms and annotating rules are not presented in the papers nor publicly accessible

• It is not possible to replicate the approaches and as a result the comparison is not feasible


Conclusions

• Applying natural language processing (NLP) techniques, SADGE automatically annotates (highlight) the architectural issues to reduce the knowledge engineering effort

• Next, a knowledge engineer manually post processes the outcome of the automated annotation process to increase the accuracy of the results

• Our preliminary evaluation shows that when the practitioners are not experienced, the automatic approach has the highest sensitivity and the lowest processing time while the semi-automatic approach has the highest accuracy

• SADGE can assist practitioners to identify required architectural decisions more confidently


Future Work

• Improving the sensitivity and accuracy of automated annotator by applying machine learning algorithms

• Increasing the training data by receiving more architectural related documents from real projects in the industry

• Evaluating the framework by conducting case studies that involve expert architects and also include project documents


References• Perez-Gonzalez, H. G.: Automatically Generating Object Models from Natural

Language Analysis, 17th annual ACM SIGPLAN conference on Object-oriented programming, systems, languages, and applications, ACM New York, USA, pp: 86 – 87 (2002)

• Bajwa, Imran Sarwar, Ali Samad, and Shahzad Mumtaz. "Object oriented software modeling using NLP based knowledge extraction." European Journal of Scientific Research 35.01: 22-33 (2009)

• Harmain, H. M. and Gaizauskas, R.: CM-Builder: A Natural Language-based CASE Tool, Journal of Automated Software Engineering, 10, 2003, pp. 157-181 (2003)

• Soeken, M., Wille, R., and Drechsler, R.: Assisted behavior driven development using natural language processing. In Objects, Models, Components, Patterns (pp. 269-287). Springer Berlin Heidelberg (2012)

• López, C., Codocedo, V., Astudillo, H., & Cysneiros, L. M. (2012). Bridging the Gap between Software Architecture Rationale Formalisms and Actual Architecture Documents: An Ontology-Driven Approach. Science of Computer Programming, 77(1), 66-80 (2012)

• Figueiredo, A. M., dos Reis, J. C., and Rodrigues, M. A.: Improving Access to Software Architecture Knowledge: An Ontology-based Search Approach. International Journal Multimedia and Image Processing (IJMIP), 2(1/2) (2012)

Thanks!

Questions?
