Upload
others
View
0
Download
0
Embed Size (px)
Citation preview
UNIVERSITY OF PATRAS
DEPARTMENT OF MECHANICAL ENGINEERING &
AERONAUTICS
DIVISION OF DESIGN AND MANUFACTURING
LABORATORY FOR MANUFACTURING SYSTEMS AND
AUTOMATION
DIPLOMA THESIS
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE
ANGELOPOULOS IOANNIS
5931
Supervisor PROFESSOR MOURTZIS DIMITRIS
Diploma thesis submitted at Department of Mechanical Engineering & Aeronautics at University
of Patras
PATRAS, July 2021
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics ii
Division of Design & Manufacturing
University of Patras, Department of Mechanical Engineering & Aeronautics ANGELOPOULOS IOANNIS © 2018 – All rights reserved
UNIVERSITY OF PATRAS
DEPARTMENT OF MECHANICAL ENGINEERING &
AERONAUTICS
DIVISION OF DESIGN & MANUFACTURING
LABORATORY FOR MANUFACTURING SYSTEMS &
AUTOMATION
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics iii
Division of Design & Manufacturing
Η παρούσα διπλωματική εργασία παρουσιάστηκε
από τον
ANGELOPOULOS IOANNIS
5931
την 15η Ιουλίου 2021
Η έγκριση της διπλωματικής εργασίας δεν υποδηλοί την αποδοχή των γνωμών του συγγραφέα.
Κατά τη συγγραφή τηρήθηκαν οι αρχές της ακαδημαϊκής δεοντολογίας.
ΣΧΕΔΙΑΣΜΟΣ & ΑΝΑΠΤΥΞΗ ΣΥΣΤΗΜΑΤΟΣ ΠΡΟΪΟΝΤΟΣ-ΥΠΗΡΕΣΙΑΣ ΒΑΣΙΣΜΕΝΟ ΣΤΗΝ ΕΠΑΥΞΗΜΕΝΗ
ΠΡΑΓΜΑΤΙΚΟΤΗΤΑ ΓΙΑ ΤΗΝ ΣΥΝΤΗΡΗΣΗ ΒΙΟΜΗΧΑΝΙΚΩΝ ΚΑΛΟΥΠΙΩΝ
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics iv
Division of Design & Manufacturing
ΠΕΡΙΛΗΨΗ
Η συντήρηση είναι ένα απαιτητικό σύνολο εργασιών που εκτελούνται σε βιομηχανικό εξοπλισμό.
Οι τεχνικοί υποχρεούνται να φέρουν έντυπα εγχειρίδια, προκειμένου να εκτελέσουν όλες τις
απαραίτητες διορθωτικές ενέργειες. Επιπλέον, υπάρχουν περιπτώσεις στις οποίες ο τεχνικός
πρέπει να επικοινωνήσει με έναν μηχανικό ή ακόμη και να καλέσει εξειδικευμένο προσωπικό για
την εκτέλεση πολύπλοκων εργασιών συντήρησης. Οι πρόσφατες εξελίξεις στις τεχνολογίες
κινητής τηλεφωνίας και στη Μικτή Πραγματικότητα (Mixed Reality), επιτρέπουν στους
μηχανικούς σε όλο τον κόσμο να σχεδιάζουν και να υλοποιούν χρήσιμες εφαρμογές για την
παροχή όλων των απαιτούμενων οδηγιών και την υποστήριξη επικοινωνίας, με τη χρήση
οποιασδήποτε έξυπνης συσκευής, βασισμένη στην προβολή τρισδιάστατου ψηφιακού
περιεχομένου στο πραγματικό περιβάλλον. Η Επαυξημένη Πραγματικότητα (Augmented Reality)
είναι μια πρωτοποριακή ψηφιακή τεχνολογία που διευκολύνει τους μηχανικούς να μειώσουν το
γνωστικό φορτίο των τεχνικών. Επιπλέον, με τη χρήση του AR μειώνεται επίσης το κατώτατο
όριο για ελάχιστες ειδικές δεξιότητες ή / και εκπαίδευση. Λαμβάνοντας υπόψη τις εξελίξεις στις
Τεχνολογίες Πληροφορίας και Επικοινωνιών (ΤΠΕ), οι ειδικοί μηχανικοί είναι σε θέση να
δημιουργήσουν τους απαιτούμενους διαύλους επικοινωνίας με τους τεχνικούς του καταστήματος,
σε μια προσπάθεια να παρέχουν εξ αποστάσεως καθοδήγηση. Ως εκ τούτου, ο συνολικός χρόνος
καθώς και το κόστος συντήρησης του βιομηχανικού εξοπλισμού μπορούν να μειωθούν περαιτέρω
καθώς και να απλοποιηθεί η διαδικασία της συντήρησης. Ο σκοπός αυτής της ερευνητικής
εργασίας είναι ο σχεδιασμός και η ανάπτυξη ενός πλαισίου απομακρυσμένης και έξυπνης
συντήρησης βασισμένης σε AR για εξοπλισμό Engineered to Order (ETO). Το framework
υλοποιήθηκε ως μια κινητή εφαρμογή AR, η οποία παρέχεται στους πελάτες από τον
κατασκευαστή σαν υπηρεσία, δημιουργώντας έτσι ένα Σύστημα Υπηρεσίας Προϊόντων (PSS).
Λέξεις κλειδιά
Επαυξημένη Πραγματικότητα, Συντήρηση, Καλούπι, Βιομηχανία, Συστήματα προϊόντων -
υπηρεσιών
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics v
Division of Design & Manufacturing
ABSTRACT
Maintenance is a demanding set of tasks performed on industrial equipment. Technicians are
obliged to carry printed manuals, in order to perform all the necessary corrective actions.
Moreover, there are cases where the technician needs to communicate with an engineer or even
call specialized personnel in order to accomplish complex maintenance tasks. Technological
advances in mobile technologies and mixed reality, have enabled engineers over the world to
produce useful applications for providing all the needed instructions and communication with the
use of any smart device, just by registering 3D content on the real environment. Augmented Reality
is a cutting-edge digital technology facilitating engineers in reducing the cognitive load of
technicians. Further to that, with the utilization of AR the threshold for minimum special skills
and/or training is also reduced. Taking into consideration the advances in Information and
Communication Technologies (ICT), expert engineers are capable of establishing sufficient
communication channels with the shopfloor technicians, in an attempt to provide to provide remote
guidance. As a result, the total maintenance time of industrial equipment can be further reduced
and simplified. The purpose of this research work is the design and development of an AR based
remote and smart maintenance framework for Engineered to Order (ETO) equipment. The
framework can be realized as a mobile AR application, which is provided to the clients from the
OEM as a Service, thus creating a Product Service System (PSS).
Keywords:
Augmented reality, Product-Service System (PSS), maintenance, Engineered-to-Order (ETO)
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics vi
Division of Design & Manufacturing
LIST OF TABLES
Table 1 Table for AR IDEs comparison ....................................................................................... 11
Table 2 Types of PSS offered and their percentages .................................................................... 17
Table 3 List of common server protection techniques .................................................................. 24
Table 4 Involved actors and their rights ....................................................................................... 30
Table 5 List of additional functionalities ...................................................................................... 44
Table 6 List of common faults for injection molds....................................................................... 57
Table 7 KPIs used for the test case ............................................................................................... 65
Table 8 Mold maintenance process steps...................................................................................... 67
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics vii
Division of Design & Manufacturing
LIST OF FIGURES
Figure 1 Industrial revolutions combined with Machine Tool and Operator Evolutions ............... 4
Figure 2 Nine (9) pillar technologies used in Industry 4.0 ............................................................. 5
Figure 3 Virtuality-Reality continuum and its variations ............................................................... 6
Figure 4 Oculus rift, a modern HMD [29] ...................................................................................... 8
Figure 5 Vuzix Augmented Reality glasses [31] ............................................................................ 9
Figure 6 Tablet used for AR application [32] ................................................................................. 9
Figure 7 AR tracking workflow .................................................................................................... 13
Figure 8 i) Typical example of Image Target used in AR applications; ii) yellow points indicate
the recognized point array for the position and pose estimation .................................................. 14
Figure 9 PSS categorization .......................................................................................................... 16
Figure 10 Percentage of PSS adoption from global firms ............................................................ 18
Figure 11 PSS system flowchart ................................................................................................... 26
Figure 12 First step in the sequence of maintenance .................................................................... 27
Figure 13 Second step in the sequence of maintenance ................................................................ 28
Figure 14 Third step in the sequence of maintenance................................................................... 28
Figure 15 Final step in the sequence of maintenance ................................................................... 29
Figure 16 System architecture - the numbers present the sequence of interactions ..................... 31
Figure 18 UML Class Diagram supporting the data modelling of the proposed framework ....... 33
Figure 19 Introductory Graphical User Interface (GUI), MMT® ................................................. 34
Figure 20 GUI for the registration of new users in the platform, MMT® ..................................... 35
Figure 21 GUI for the login of registered users, MMT® .............................................................. 36
Figure 22 Client's start page GUI, MMT® .................................................................................... 36
Figure 23 PSS customizer interface, MMT® ................................................................................ 37
Figure 24 Date picker interface, MMT® ....................................................................................... 37
Figure 25 Customer information sheet, MMT® ............................................................................ 38
Figure 26 Engineer's start page, MMT® ....................................................................................... 39
Figure 27 Download file interface, MMT®................................................................................... 39
Figure 28 BOP creation GUI, MMT® ........................................................................................... 40
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics viii
Division of Design & Manufacturing
Figure 29 XML file structure for the generation of BOP, and generation of AR instructions ..... 40
Figure 30 Example XML file based on the structure presented previously ................................. 41
Figure 31 Mold-list GUI ............................................................................................................... 42
Figure 32 Shop Inspection GUI, MMT®....................................................................................... 43
Figure 33 Technician's Start Page, MMT® ................................................................................... 43
Figure 34 3D presentation interface, MMT® ................................................................................ 44
Figure 35 Explode functionality in 3D presentation, MMT® ....................................................... 45
Figure 36 Annotation functionality in 3D presentation, MMT®................................................... 46
Figure 37 Annotation on 3D component, MMT® ......................................................................... 46
Figure 38 GUI for Report creation & submission to the Cloud platform, MMT® ....................... 47
Figure 39 Reports, saved on the Cloud FTP server ...................................................................... 48
Figure 40 AR visualization of the exploded assembly ................................................................. 49
Figure 41 AR mold projection, health monitoring........................................................................ 50
Figure 42 Cloud database components ......................................................................................... 51
Figure 43 Typical paradigm of PHP script ................................................................................... 52
Figure 44 Administrator interface, MMT® ................................................................................... 53
Figure 45 Administrative tools interface, MMT® ......................................................................... 53
Figure 46 The mold manufacturing company ............................................................................... 55
Figure 47 Injection mold cut-out .................................................................................................. 56
Figure 48 Injection mold cycle with the corresponding times ...................................................... 57
Figure 49 Injection mold common failures ................................................................................... 58
Figure 50 Application of the developed framework ..................................................................... 62
Figure 51 Workflow of the validation scenario of first prototype of the tool. .............................. 68
Figure 52 Tool evaluation results ................................................................................................. 69
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics ix
Division of Design & Manufacturing
ABBREVIATIONS
3D 3 (Three) Dimensional
AEC Architecture, Engineering & Construction
AR Augmented Reality
BOM Bill of Materials
BOP Bill of Processes
CAD Computer Aided Design
CBM Condition Based Maintenance
CNC Computer Numerical Control
ETO Engineered to Order
FTP File Transfer Protocol
GUI Graphical User Interface
HMD Head Mounted Display
ICT Information and Communication Technology
ID Identification
IDE Integrated Development Environment
IM Injection Mold
IPS2 Industrial Product Service Systems
LAN Local Area Network
MR Mixed Reality
MRI Magnetic Resonance Imaging
OEM Original Equipment Manufacturer
OHMD Optical Head Mounted Display
PBH Pay By the Hour
PSS Product-Service System
QR code Quick Response code
R&D Research and Development
SDK Software Development Kit
VE Virtual Environments
VR Virtual Reality
XML Extensible Markup Language
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics x
Division of Design & Manufacturing
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics xi
Division of Design & Manufacturing
CONTENTS
ΠΕΡΙΛΗΨΗ................................................................................................................................. IV
ABSTRACT .................................................................................................................................. V
LIST OF TABLES ...................................................................................................................... VI
LIST OF FIGURES .................................................................................................................. VII
ABBREVIATIONS ..................................................................................................................... IX
CONTENTS................................................................................................................................. XI
CHAPTER 1. INTRODUCTION ........................................................................................... 1
CHAPTER 2. LITERATURE REVIEW ............................................................................... 3
2.1 4TH INDUSTRIAL REVOLUTION ................................................................. 3
2.2 VIRTUAL ENVIRONMENTS (VE).................................................................. 5
2.3 FIELDS OF APPLICATION ............................................................................. 6
2.4 HARDWARE USED IN AUGMENTED, MIXED, AND VIRTUAL
REALITY .............................................................................................................................. 8
2.5 SOFTWARE USED IN AR ................................................................................. 9
2.6 USER TRACKING MODEL ............................................................................ 12
2.7 PRODUCT-SERVICE SYSTEMS (PSS) ........................................................ 15
2.8 DATA PROTECTION AND DATA ENCRYPTION .................................... 23
CHAPTER 3. PROPOSED METHODOLOGY AND SYSTEM ARCHITECTURE25
3.1 PSS MODEL ...................................................................................................... 25
3.2 SYSTEM ARCHITECTURE ........................................................................... 29
3.3 DATA FORMALIZATION .............................................................................. 32
3.4 APPLICATION FUNCTIONALITIES ........................................................... 34
CHAPTER 4. INDUSTRIAL TEST CASE ......................................................................... 54
4.1 INJECTION MOLD OVERVIEW .................................................................. 56
4.2 CASE STUDY DESCRIPTION ....................................................................... 60
CHAPTER 5. RESULTS – KEY PERFORMANCE INDICATORS ............................... 63
5.1 KEY PERFORMANCE INDICATORS.......................................................... 63
5.2 TEST CASE CURRENT SITUATION DESCRIPTION .............................. 63
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics xii
Division of Design & Manufacturing
5.3 EXPECTATIONS OF ADOPTING THE PROPOSED METHODOLOGY
64
5.4 VALIDATION METHODOLOGY ................................................................. 66
5.5 RESULTS ........................................................................................................... 68
CHAPTER 6. CONCLUDING REMARKS AND OUTLOOK ......................................... 70
ACKNOWLEDGEMENT .......................................................................................................... 72
PUBLICATIONS RELEVANT TO THE THESIS ................................................................. 72
REFERENCES ............................................................................................................................ 74
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 1
Division of Design & Manufacturing
CHAPTER 1. INTRODUCTION
Manufacturing is a system in which product design is the initial stage and the delivery of finished
products to the market is the final output. Manufacturing can also be described as a whole of
decisions through the processes. Generally, there are four (4) classes of manufacturing attributes
to be considered during decision making: cost, time, quality, and flexibility. Bearing also in mind
that manufacturing has moved from the mass production era, to the mass customization era,
engineers around the world have to constantly innovate, in order to keep the competitive edge in
their field of operation [1]. Augmented Reality is a powerful tool, that can help minimizing
production costs and times while ensuring the quality of the products [2].
Augmented Reality is an emerging technology which allows the user to implement computer-
generated content on the physical, real-world environment using suitable equipment. The content
displayed can be of any type: images, 3D models, animations, videos, and sound. The key feature
of AR is that the user is not obliged to fully immerse on a virtual environment. Although AR is
considered to be on a premature level, is a quite old technology. The first idea of a virtual
environment was made by L. Frank Baum (character maker). The first application of such
technology was developed by Morton Heilig with a simulator called Sensorama [3]. However, MR
is subject to constant development as users’ needs constantly increase by the time. At this point, it
is commonly accepted that equipment used in such applications is at its second generation.
In the past two decades, Augmented Reality (AR) has received a growing amount of attention by
researchers in the manufacturing technology community, because AR can be applied to address a
wide range of problems throughout the assembly phase in the lifecycle of a product, namely
planning, design, ergonomics assessment, operation guidance and training [4].
The major question to all the MR developers is what is the purpose of building and using such
systems? The answer is simple. MR systems in AEC aim to help users overcome difficulties while
performing certain tasks (e.g. help on-sight worker identifying a system failure with the help of a
master engineer in real-time), reduce the required cognitive level (e.g. help workers perform
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 2
Division of Design & Manufacturing
maintenance tasks on complex systems without the need of special knowledge or even carrying
printed instruction manuals).
AR as mentioned is an immature technology although it exists (even as an idea) for many years.
The last two decades has met great advance and still goes on with the technological advances
contributing to its advance. AR applications cover a wide variety of user needs. The most common
applications are: (a) industrial / manufacturing, (b) military, (c) entertainment and (d) education.
The purpose of this thesis is to address some issues related to an industrial use case. More
specifically, AR is nice-to-have tool in maintenance industry. Maintenance personnel often need
to dismantle complex mechanisms and/or in some cases there are hidden components playing a
dramatic role in the maintenance process. The traditional solution is to refer to paper-printed
manuals, communicated via telephone with a master engineer or in the worst case, report the failure
to the manufacturer, and then a highly trained worker is sent to the sight. The above situations
present a whole of disadvantages, for both clients and manufacturers. First of all, maintenance
costs can rise very high, a matter which is amplified by the risk included (mainly during the fault
diagnosis and during the repair tasks). Secondly it is time consuming, as inexperienced workers
try to sort out the fault and explain it to the master engineer, providing limited info. Especially in
the case of sending a highly trained worker from the manufacturer, machinery down-times rise in
catastrophic rate for the production line [5].
The framework proposed in this thesis can be realized as an AR application prototype that aiming
to enable workers perform maintenance tasks without the need of carrying printed manual
instructions. Concretely, for the utilization of the AR application, technicians carry a smart device,
such as a smart phone or tablet, which contains all the steps and precautions in vivid 3D/AR
presentations accompanied by text instructions. Moreover, if communicating with a master
engineer is inevitable, the features of such a smart device (camera, phone calls, connection to
worldwide web, etc.), help worker share the proper amount of information with the engineer, who
in the best-case scenario will be able to produce real-time instructions via video streaming.
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 3
Division of Design & Manufacturing
CHAPTER 2. LITERATURE REVIEW
Modern industrial equipment and machinery are composed of complex systems and delicate
components. Those systems and components often break down from excessive use or other
unpredictable factors. Thus, the need for a cluster of maintenance tasks is mandatory. Moreover,
the aim on modern maintenance scheduling is reducing or even eliminating, if possible,
downtimes. A solution to this problem lies on Condition Based Maintenance (CBM). CBM is a
maintenance strategy that monitors the actual condition of the asset to decide what maintenance
needs to be done. CBM dictates that maintenance should only be performed when certain
indicators show signs of decreasing performance or upcoming failure. Checking a machine for
these indicators may include non-invasive measurements, visual inspection, performance data and
scheduled tests. Condition data can then be gathered at certain intervals, or continuously (as is
done when a machine has internal sensors). The goal of CBM is to spot upcoming equipment
failure so maintenance can be proactively scheduled when it is needed. Asset conditions need to
trigger maintenance within a long enough time period before failure, so work can be finished
before the asset fails or performance falls below the optimal level. An advanced condition
monitoring solution can be necessary to keep product quality high and scrap low in the injection
molding process.
2.1 4TH INDUSTRIAL REVOLUTION
The last three centuries, the Industrial landscape has undergone a series of changes which have
improved the efficiency of manufacturing processes and manufacturing systems, have enabled
manufacturers to integrate customers in the development and production of new products,
companies have improved their business strategies and finally, through the collaboration of
different parties the creation of advanced manufacturing networks has become reality. The above-
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 4
Division of Design & Manufacturing
mentioned are enabled/facilitated by the integration of a plethora of technological advances, which
can be clustered and summarized in the so-called Industrial Revolutions. Up to today, a total of
four Industrial Revolutions have occurred. In Figure 1, the Industrial Revolutions are illustrated
based on their chronological appearance [6-9]. What is interesting in this figure, is the evolution
of manufacturing personnel as a result of the technological advances, which are indicated as
Operator 1.0-5.0. Therefore, it becomes evident, that maturity, knowledge, and experience of
technicians has been upscaled, enabling humans collaborate with machines and by extension
improved productivity rates, achieve better quality, minimize errors, etc.. The current situation in
the manufacturing domain can be described as the fourth industrial revolution, better known as
Industry 4.0. Furthermore, this term refers to the wave of change the industrial world is facing, the
last five years. It is originated by Germany has proven to be at the top competitors in the fields of
R&D, production of manufacturing systems and management of industrial processes [10].
Figure 1 Industrial revolutions combined with Machine Tool and Operator Evolutions
Following the recent developments regarding new digital industrial technologies, under the
framework of Industry 4.0, the collection and data analysis across machines is enabled / facilitated,
thus allowing for the complete digitalization of modern manufacturing systems. Consequently, the
production of higher quality at reduced costs in faster, more flexible, and more efficient processes.
As a result it is expected that the above-mentioned advances will boost productivity rates, improve
the economic efficiency of manufacturing processes, promote industrial growth, as well as they
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 5
Division of Design & Manufacturing
will also change the key characteristics of human workforce. By extension the competitiveness of
manufacturing companies can be improved. Taking into account recently published research
works, it can be concluded that cutting-edge digital technologies are increasingly being integrated
in the manufacturing domain, providing a strong indication that the manufacturing domain is
transformed [11-16]. Moreover, it will lead to higher efficiencies and alter traditional supplier,
manufacturer, and customer production relationships, as well as human and machine relationships.
The building blocks of Industry 4.0 can be summarized into the nine technology trends presented
in Figure 2 [17].
Figure 2 Nine (9) pillar technologies used in Industry 4.0
2.2 VIRTUAL ENVIRONMENTS (VE)
Virtual environments are one of the latest technological advances. Combined with the increasing
commercial availability of such equipment, virtuality has become the latest trend. However, these
technologies can be used in modern industry in order to facilitate the automation of processes and
speed up production. For this purpose, a number of different implementations can be integrated in
modern industrial business models, in order to provide more developed product services to
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 6
Division of Design & Manufacturing
customers, which in turn create added value for the manufacturers, thus increasing their revenue
and extend product lifecycle. The most common variations of VE found in modern industry, are
virtual reality, augmented reality, and holograph based implementations. In this thesis, the
proposed application is based on the implementation of Augmented Reality. Augmented Reality
is the technology, which enables the projection of 3D virtual objects onto the physical
environment. In contrast with virtual reality, the user of AR is not fully immersed in a virtual
environment, he/she maintains his/her contact with the real world, which is where the virtual
objects are superimposed [18].
Figure 3 Virtuality-Reality continuum and its variations
Augmented Reality is used extensively in maintenance applications. Not only that, but AR
applications can also be found in projects where visualization of warehouse or production line
status is needed [10,19,20]. Following the same concept augmented reality can also be used in
conjunction with data obtained from sensors attached to the equipment in order to monitor
condition. Such functionality can be proved to be useful in predictive maintenance.
2.3 FIELDS OF APPLICATION
Despite being around for a short period of time, augmented reality has invaded many scientific
fields. In some fields, it is used as a mean of impressing and promoting. Some other fields use it
for prototype visualization in a cheaper, portable, intriguing, and flexible way while others use it
as a tool for continuous information flow to an operator for performing complex tasks.
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 7
Division of Design & Manufacturing
Military: Military research groups, have developed AR applications for troop support on the field
or for remote maintenance. It is fact that military conducts many scientific researches, while most
of them are of great interest, due to the higher quality militaristic needs and due to great
investments. Military seeks, new technologies that could be implemented on weapon systems, in
order to increase their efficiency. In 1993, Rockwell International created video map overlays of
satellite and orbital debris tracks to aid in space observations at Air Force Maui Optical System.
Such systems are also used in telescopes, in order to visualize satellite and space debris trajectories
[21].
Medical: The use of new technologies in medical applications is limited, mainly due to the fact
that both doctors and patients are reluctant to try innovative methods. Despite the rapid
development, especially in the usage of robot for higher accuracy in surgeries, lack of knowledge
and increased cost has diminished its growth. Augmented Reality can be used in this field as a tool
by the doctor for data flow and disease diagnosis with relatively low cost. More specifically, a
wide variety of applications has been developed so as to allow the doctor-operator to have a
continuous image derive from an ultrasound, a Computed Tomography, or a Magnetic resonance
imaging (MRI) scan [22].
AEC: For at least a decade, several AEC researchers have promoted AR technology adoption
because of its potential as a visualization aid. Roberts et al. [23] developed an AR system to render
images of underground structures onto a view of the site. Hammad et al. [24] and Thomas et al.
[25] found the potential benefits of AR in infrastructure field tasks and architectural assembly
guidance, respectively. Some researchers demonstrated AR systems for planning or design process
such as design detailing [26], outdoor architectural designs, and urban planning [27]. Behzadan
and Kamat [28] proposed an AR system for AR rendering of construction processes onto the
construction site.
Advertising/ Promotion: Advertisers, always try new things and technologies, in order to get
shoppers’ attention. The technological era world is going through, is strictly connected to digital
information. Moreover, smart phones have become a necessity to modern peoples’ life. Thus, AR
is a brilliant tool to advertise and promote new products, without the need to spend huge amounts
on tangible commercial material (e.g. flyers) or promote new products using outdated
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 8
Division of Design & Manufacturing
technologies. A great example is the use of QR coding system, which in fact is a two-dimensional
barcoding system (matrix barcode).
2.4 HARDWARE USED IN AUGMENTED, MIXED, AND VIRTUAL REALITY
Head Mounted Displays (HMD): A head mounted display, is a display device, worn on the user’s
head or as part of a helmet, that has a small display optic in front of on (monocular HMD) or each
eye (binocular HMD). There is also an optical head mounted display (OHMD), which is a wearable
display that has the capability of reflecting projected images as well as allowing user to see through
it.
Figure 4 Oculus rift, a modern HMD [29]
AR glasses: Vuzix is one of the leading companies in the AR glasses industry. The lab is equipped
with a pair of Vuzix Star 1200 XL glasses [30], which have performed satisfactorily under various
circumstances. After many hours of testing and different projects they are used in, two major
drawbacks have emerged, which have to be taken into consideration for an industrial AR use case,
where those glasses would have a primary role. The first drawback is the limited mobility due to
the cables which allow the glasses to be connected to the computer. Cables, limit worker’s
maneuverability, thus working in tight spaces or working around machinery may require worker
to remove glasses, or even worse carry the computer (laptop) in a backpack. The second drawback
is the weight of the glasses themselves. These glasses are too heavy creating fatigue if used for
long hours. Especially, if working in hot sites, sweat makes the glasses slip on the worker’s face
although they have an adjustable strap. However, Vuzix seems to have encountered those details
and below follows a list with their latest models and comparison of their key features.
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 9
Division of Design & Manufacturing
Figure 5 Vuzix Augmented Reality glasses [31]
Smart Devices: A smart device is an electronic device, capable of connecting to other devices or
networks via different wireless protocols (e.g. Bluetooth, Wi-Fi, 3G etc.) and operating to some
extent interactively and autonomously. Some types of smart devices are smartphones, tablets,
smartwatches, and smart bands. Such devices can be complementary used in MR applications,
where additional user input is needed or can be used to solely implement an AR application on
them. This type of devices is currently preferred among the others, due to their compactness, their
mobility and the fact that are not purpose oriented. The latter, meaning that one can use a smart
device as a general tool with which he/she can implement a MR application, without the need of
any dedicated equipment.
Figure 6 Tablet used for AR application [32]
2.5 SOFTWARE USED IN AR
Besides the hardware described in the previous section, specific software is used in order to
develop augmented reality applications. In this section, all the aspects of an AR application,
regarding software are discussed. An AR application consists of scenes. These scenes consist of
3D models, text etc. and scripts.
The 3D models are produced from 3D CAD software or other similar software. If needed dedicated
software is used to convert the 3D geometries into a convenient filetype (.fbx, .obj, .dae, .3ds, .dxf
and .skp) [33-35].
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 10
Division of Design & Manufacturing
For the development of AR applications there are several development kits on the market available.
Having a list with the predominant SDKs available in the market, the most suited has to be chosen.
The criteria based on which the choice will be made, are overall performance, cost, and support.
In Table 2, which has been adopted from reference [36], the most common AR SDKs are listed
and analyzed based on the above-mentioned criteria. Taking into consideration these criteria the
Vuforia SDK has been chosen [37]. The reasoning supporting this decision is provided briefly in
the following paragraph.
Although, from the comparison presented in Table 1, Wikitude gets the highest score, Vuforia is
the chosen SDK, which comes 2nd, having better features (maximum distance capturing,
recognition stability, minimum angle recognition), features that are thought to be more crucial
when developing an industrial AR based application. Another reason that led to choosing Vuforia
over the other SDKs, is support. Vuforia, is mating with Unity 3D™ [38] seamlessly and there is
plenty support from online users. Moreover, working in similar projects in the past, has gained a
considerable amount of knowledge concerning the use of Vuforia in Unity 3D™.
As stated above, the production of AR scenes requires a series of scripts. Scripts are written
instructions on a programming language, which instruct computer how to act in all situations. The
programming language chosen is C# (C-sharp). The choice of programming language was made
based on internet support and material, on the fact that it can be used for PC and Android
development and finally because it is one of the three supported programming languages, Unity
3D supports. In order to write scripts, one needs a suitable Integrated Development Environment
(IDE). Unity 3D is supported by the MonoDevelop IDE. However, in order to extend the
functionality of the proposed framework and the AR application, the Visual Studio IDE [39] has
been selected. Further to that, the Visual Studio IDE has been selected due to the fact that it is
more stable and offers advanced debugging capabilities.
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 11
Division of Design & Manufacturing
Table 1 Table for AR IDEs comparison
SDK
Feature
description
Vuforia EasyAR Wikitude ARToolKit Kudan MaxST Xzimg NyARToolKit
Maximum
distance capturing
/ holding marker
(m)
1.2 / 3.7 0.9 / 2.7 0.8 / 3 3 / 3 0.8 / 3 0.5 / 0.9 0.7 / 5 0.7 / 1
Recognition
stability of
immovable marker
10 7 6 8 10 7 8 5
Recognition
stability of
movable marker
6 3 4 6 6 2 7 3
Minimum angle recognition
30o 35o 40o 10o 30o 50o 35o 45o
Minimum
visibility for
recognition
overlapped marker
20% 10% 30% 100% 25% 50% 10% 75%
2D Recognition ✓ ✓ ✓ ✓(bordered) ✓ ✓ ✓ ✓
3D Recognition ✓ - ✓(beta) - ✓ ✓ - -
Geo-Location - - ✓ - - - - -
Cloud Recognition ✓ - ✓ - - - - -
SLAM* - - ✓ - ✓ ✓ - -
Total (rating) 7.1 4.4 7.5 2.8 6.9 5.2 4.7 3.1
*SLAM stands for Simultaneous Localization and Mapping and is a technology which enables the
understanding of the physical world through a set of feature points [37].
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 12
Division of Design & Manufacturing
2.6 USER TRACKING MODEL
Tracking is a crucial part in AR as it positions the virtual objects in the real environment of the
user [40,41]. To achieve tracking, it uses the camera sensor integrated to the device for the
recognition of predefined frame markers which are placed within the field of view of the user. The
transformation T between a camera and a marker is:
𝑥𝑥𝑐𝑐 = 𝑇𝑇 ∗ 𝑋𝑋 (1)
Where: X is a point in world coordinates, xc is its projection in ideal image coordinates and T is
the pose matrix. Transformation T consists of translation vector t and 3 x 3 rotation matrix R:
𝑥𝑥𝑐𝑐 = [𝑅𝑅|𝑡𝑡] ∗ 𝑋𝑋 (2)
which, using homogeneous coordinates becomes:
�𝑥𝑥𝑦𝑦𝑧𝑧� = �
𝑟𝑟1 𝑟𝑟2 𝑟𝑟3 𝑡𝑡𝑥𝑥𝑟𝑟4 𝑟𝑟5 𝑟𝑟6 𝑡𝑡𝑦𝑦𝑟𝑟7 𝑟𝑟8 𝑟𝑟9 𝑡𝑡𝑧𝑧
� �
𝑋𝑋𝑌𝑌𝑍𝑍1
� (3)
In addition to that, in order to map between frame marker ideal image (xc) and observed pixel
coordinates (xpix) a camera calibration matrix K is used:
𝑥𝑥𝑝𝑝𝑝𝑝𝑥𝑥 = 𝐾𝐾 ∗ 𝑥𝑥𝑐𝑐 → �𝑥𝑥𝑝𝑝𝑝𝑝𝑥𝑥𝑦𝑦𝑝𝑝𝑝𝑝𝑥𝑥
1� = �
𝑓𝑓 0 𝑝𝑝𝑥𝑥 00 𝑓𝑓 𝑝𝑝𝑦𝑦 00 0 1 0
� �𝑥𝑥𝑐𝑐𝑦𝑦𝑐𝑐𝑧𝑧𝑐𝑐� (4)
As a result, when the camera detects the frame marker,
𝑥𝑥𝑝𝑝 = 𝐾𝐾 ∗ 𝑇𝑇 ∗ 𝑋𝑋𝑝𝑝 (5)
Where xi are the positions of the four corners (i=1,2,3,4) of the marker in the camera image and Xi
are their corresponding world coordinates. Thus,
�𝑥𝑥𝑝𝑝𝑦𝑦𝑝𝑝1� = �
𝑓𝑓 0 𝑝𝑝𝑥𝑥 00 𝑓𝑓 𝑝𝑝𝑦𝑦 00 0 1 0
� �
𝑟𝑟1 𝑟𝑟2𝑟𝑟4 𝑟𝑟5
𝑟𝑟3 𝑡𝑡𝑥𝑥𝑟𝑟6 𝑡𝑡𝑦𝑦
𝑟𝑟7 𝑟𝑟80 0
𝑟𝑟9 𝑡𝑡𝑧𝑧0 1
� �
𝑋𝑋𝑝𝑝𝑌𝑌𝑝𝑝𝑍𝑍𝑝𝑝1
� (6)
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 13
Division of Design & Manufacturing
The system then calculates M = K * T,
�𝑥𝑥𝑝𝑝𝑦𝑦𝑝𝑝1� = �
𝑚𝑚1 𝑚𝑚2 𝑚𝑚3 𝑚𝑚4𝑚𝑚5𝑚𝑚9
𝑚𝑚6𝑚𝑚10
𝑚𝑚7 𝑚𝑚8𝑚𝑚11 𝑚𝑚12
� �
𝑋𝑋𝑝𝑝𝑌𝑌𝑝𝑝𝑍𝑍𝑝𝑝1
� (7)
which is the final transformation matrix and applies it to all visualized geometries, in order to
estimate the position of the digital visualizations on the virtual environment. With the aim of
creating a system whose tracking is accurate and robust, PTC Vuforia SDK was selected. It is a
commercial product that provides an extension for Unity 3D™, for creating AR applications. An
illustrative example of the AR tracking algorithm based on the model discussed above is presented
in the following figure (Figure 7).
Figure 7 AR tracking workflow
At this point, it is stressed out that the above mentioned mathematical model for the position and
pose estimation of the user within an Augmented Environment can be adjusted by the addition of
the following models in order to be implemented on Mixed Reality equipment, such as the
Microsoft Hololens (version 1, and 2). More specifically, the model has to take into account the
feedback from the four (4) integrated depth sensors / cameras. Consequently, the following
equation for the depth calculation is implemented:
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 14
Division of Design & Manufacturing
𝐷𝐷 =𝑅𝑅
√𝑈𝑈2 + 𝑉𝑉2 + 1 (8)
Where:
• D is the calculated depth
• R denotes the range, as measured from the HMD ToF (Time-of-Flight) camera
• U,V are the measured values for distance of a specific, user-defined point in the physical
environment
As a result, the interpretation of the depth value in real-world coordinates is derived from Equation
9:
�𝑥𝑥𝑦𝑦𝑧𝑧� = 𝐷𝐷�
𝑈𝑈𝑉𝑉1� (9)
Next, in Figure 8, a common Image Target used for the development and implementation of AR
applications is presented. Concretely, on the left side of the figure, the Image Target is illustrated
in its physical form, whereas in the right side, the points recognized by the sensing system of the
AR device have been highlighted for reader-reference [42].
Figure 8 i) Typical example of Image Target used in AR applications; ii) yellow points indicate the
recognized point array for the position and pose estimation
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 15
Division of Design & Manufacturing
2.7 PRODUCT-SERVICE SYSTEMS (PSS)
Product Service Systems (PSS) is a term which was introduced in the early 60s by Bristol Siddeley
as a system of engines followed by support services (e.g. maintenance). However, the initial idea
was named differently (Pay by the hour, PBH), it is the predecessor of PSS. Goedkoop M. in [43],
defines PSS as the combination of product(s) and service(s) in a system to deliver the required user
functionality in a way that reduces the impact on the environment and moreover on states and
governments.
A PSS values asset performance or utilization rather than ownership and achieves differentiation
through the integration of product and services that provide value in use to the customer. PSS and
IPS2, can be categorized. The three main categories in which all product service systems are
included are, Product-Oriented Services, Use-Oriented Services and Result-Oriented Services.
Product-Oriented services is the kind of services in which customers get full ownership of the
tangible product while they benefit from the integration of a series of services (e.g. maintenance
contracts).
Use-Oriented services is a kind of services in which the service provider retains ownership of the
tangible product. In this type of services customer benefits from the function of the products.
Examples of these services are leasing, pooling, and sharing.
Result-Oriented services is the type of services in which, although, the customer and the supplier
agree on a result there is no product pre-determined [44].
Many different categorizations for PSSs have been introduced. The approach given above is about
the main categories and it must be mentioned that they are globally accepted. The other approaches
are about further categorization, like the categorization shown below [44].
Over the years the global market changes, as a result demands to manufacturers have increased
leading to the development and implementation of product service systems (PSS). PSS refers to
services (usually intangible) that support tangible products. They offer strategic innovation which
companies use in order to separate resource consumption from its link to profit and standard of
living improvements, to find new profit centers, to compete and generate value and social quality
while decreasing total resource consumption. Generally, PSS can be considered as a win – win
solution for both suppliers and customers. A special category of PSS is those that refer to business
products (Business to Business or B2B). That kind of services is called Industrial Product Service
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 16
Division of Design & Manufacturing
Systems (IPS2). Also, IPS2 provide strong opportunities for business innovation and sustainability
improvement. In many researches, we see that IPS2, are aiming in higher sustainability of the
products they support. The relation between IPS2 and sustainability is a two dependency. As stated
in [45] modern markets shift their focus to sustainability and ecological efficiency of products.
From the former statements, one can easily understand the importance of sustainability in IPS2 and
generally in PSS. Sustainability can be implemented by establishing closed loop recycling
management with reuse of services. Another aspect of sustainability is prolonged life cycles by
preserving the usability of the IPS2 offers.
Technological evolution has a great impact on IPS2. Having more complex machines on the
market, the need for IPS2, and moreover for advanced IPS2, is mandatory. On the other hand,
advances in technological aspects, such as wireless data transmission, mobile devices, information,
and communication technology (ICT) [46], give opportunities for more advanced and satisfactory
IPS2, that meet the demands of the modern market.
The aim of integrating IPS2 is to increase economic competitiveness by selling functionality
instead of selling products. However, IPS2 is a suitable solution for companies that have positioned
themselves as niche players (e.g. agricultural engineering, packing). In order to provide more
sophisticated IPS2, suppliers have to take into consideration that by implementing IPS2 to their
tangible products, their relationship with the customer changes to an integrative cooperation.
Figure 9 PSS categorization
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 17
Division of Design & Manufacturing
In [47] we can extract useful information about the impact of IPS2 in the global market, as well as
clarify how beneficial their implementation is for both suppliers and customers.
Below follows a table which shows the most common types of IPS2 and also analyses the
percentages and number of companies that provides them, from a sample of 12521 firms. The
sample consists of 29.52% combined manufacturing enterprises and 68.70% pure manufacturing
enterprises. Table 2 Types of PSS offered and their percentages
Which services are offered % of firms offering service Number of firms offering
service
Consulting Services 2.69% 291
Design & Development
Services
21.92% 2723
Financial Services 3.89% 421
Installation &
Implementation Services
5.10% 552
Leasing Services 1.07% 116
Maintenance & Support
Services
11.94% 1293
Outsourcing & Operating
Services
1.68% 182
Procurement Services 1.15% 125
Property & Real Estate 3.83% 415
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 18
Division of Design & Manufacturing
Retail & Distribution
Services
12.18% 1319
Systems & Solutions 15.70% 1700
Transportation & Trucking
Services
0.20% 22
By the interpretation of the information presented in Table 3, it can be safely concluded that among
the top three (3) IPS2 are the Design & Development services, the Systems & Solutions services,
and the Retail & Distribution services.
Figure 10 Percentage of PSS adoption from global firms
The above chart is featured in [3]. It shows the percentage of enterprises from various markets
around the world that have implemented IPS2 on their products. It is clear that western markets
have shifted their focus on IPS2, while eastern markets have not yet. This can be useful as engineers
have a clearer view on the global market needs. However, in order to equalize the bars on the
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 19
Division of Design & Manufacturing
former chart, engineers have to examine each market (what manufacturers produce and the target
groups) individually, in order to provide optimum solutions.
More and more companies around the world provide PSS packages. Some of the firms are solely
providing PSS while others manufacture tangible products and integrate services on them.
The idea of a company that provides only services for equipment bought from another firm is very
beneficial. Firstly, customer can retain their management model, or in some cases adapt it keeping
costs at low level. Secondly, service companies, because of their nature can provide services of
higher quality as it is their one and only field of expertise. Thirdly, with the creation of such
companies, new positions for jobs are created. However, the only downside in the whole idea is
the level of integration in the customer’s company.
The most common IPS2, found in today’s markets are:
• Maintenance
• Repairing
• Training
• Retrofitting/Customizing/Upgrading
• OEM replacement parts distribution
• Machine tool calibration
• Emergency services
• Plant installation/relocation
• Customer requested services/Design
• Safety Services
• Leasing/renting equipment/machinery
• Data acquisition and analysis
Machine tool calibration (metrology/laser calibration): Modern machines are usually CNC
which means that they incorporate many measuring components, which are electronically
controlled. Over time these machines tend to “lose” their calibration/ mapping, resulting in faulty
products, or in most severe cases damage in machine components (e.g. if a mill head gets out of
control may exceed its movement limits). Companies like Mori Seiki, API Services and MTT
Machine Tool Technologies, provide such services utilizing experienced personnel. We have to
note, that Mori Seiki is a manufacturer and has implemented that type of service to their machines.
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 20
Division of Design & Manufacturing
The other two firms (API Services and MTT Machine Tool Technologies) are exclusively service
companies.
Training: Manufacturing is composed of many demanding processes, so is the machinery used.
Taking into consideration the technological advances, it is easily understandable, that operating
machinery is a very difficult process, and moreover getting the most of it requires the best
knowledge of its usage, otherwise the workflow is not optimal and the safety of the on-sites
workers is put under great risk. In these terms, mostly manufacturers and less service companies,
have integrated their products with training services, in order to help customers, organize their
plant optimally, increase their productivity and reduce emissions and resource usage by operating
the machinery in the correct way. Such companies are Machine Tool Technologies (MTT), API
Services, Mori Seiki, Tetra Pak, Sitec Industrietechnologie GmbH.
OEM replacement parts: Modern machines consist of a large number of components. Some of
them are complicated and others are simple and are treated as consumables. Either way, a business
willing to keep its competitive edge, needs to maintain their machinery in the best possible
condition. Also, if an unexpected damage occurs, minimum down time and correct fitment parts
are highly required. The best way to satisfy all the former conditions is to use OEM consumables
and other replacement parts. In this type of service, we meet only manufacturers, as they tend to
keep a large stock of all the components and consumables used in their products. Ideally immediate
support is required (within 24 hours the components are sent to the customer). Firms like Niles
Simmons, Mori Seiki, Tetra Pak, and Sitec Industrietechnologie GmbH keep a sufficient stock of
their components and they state that they ship them to the customer within 24 hours.
Maintenance/Repairing – Retrofitting*/Upgrading/Customizing: As stated above keeping
machinery to its best condition, maximum results can be achieved in terms of productivity,
energy/resources consumption and profit. Although using OEM equipment is highly suggested,
servicing, diagnosing, overhauling, and maintaining machinery is equally demanding process to
operating them. Thus, manufacturers and service companies provide their expertise to their
customers in order to ensure that the jobs done on customers’ machinery are done the best way
possible. From the customers’ side, such services are useful as they eliminate the risk of a wrong
job (resulting in further damage, longer downtime, safety issues etc.) and relief them from the cost
of training personnel specifically for that kind of tasks.
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 21
Division of Design & Manufacturing
*Retrofitting is the addition of new technology/features to older systems and power plants,
focusing on improving power plant efficiency, increase output and reduce emissions [48].
Emergency services: It is widely known that industries, in order to maximize their productivity,
they work 24/7/365. As a result, equipment wears out much faster and sometimes unexpectedly.
Thus, customers need immediate support, to diagnose the problem, if it is a simple fix to repair it
and/or if possible to have constant communication with the supplier in order to get instructions on
how to repair the broken machine. Manufacturers and service companies for that reason, maintain
hot lines 24/7/365, in order to help their customers in every possible way, such firms are MTT
Machine Tool Technologies, EMCOR Group [49], Sitec Industrietechnologie GmbH [50].
Plant installation/relocation: Achieving the best factors for productivity, product life cycle,
emissions and resources control, workflow, and profit, can be done using various techniques,
models, and services. Here stating the obvious, we present plant installation or even relocation
after a new purchase/customization. Customers are not supposed to know the best way to install
their equipment, in the case of a new machine special treatment may be needed or if the product
produced may need special environmental conditions (e.g. food industries). Thus, companies like
MTT Machine Tool Technologies, Williams Industrial Services Group LLC, EMCOR Group,
Sitec Industrietechnologie GmbH, and Tetra Pak [51], provide their expertise at the service of their
customers. Expanding the term of installation and relocation, service companies like Williams
Industrial Services Group LLC, EMCOR Group and ISG Inc. offer extended services on building
the whole plant, the piping, the vessels and generally do all the tasks to utilize the customer’s
premises optimally.
Customer requested services: From the previous paragraphs it can be safely concluded that both
manufacturers and service companies, have implemented a wide range of services, covering almost
every customer. However, if customers discover a need that is not covered by any available service,
they can refer to EMCOR Group, Williams Industrial Services Group LLC, ISG (Industrial Service
Group Inc.) and other firms similar to them, in order to cooperate and find a viable solution/design
a new service (or series of services) in order to cover the void.
Safety services: Industrial environments because of their nature are very dangerous places to work
in. Especially in cases where resources are harmful not only for the environment but also for the
on-site workers. Although protective meters are of great importance, there are some services that
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 22
Division of Design & Manufacturing
ensure safety under all situations (e.g. special insulating of components & machines, thorough
cleaning of walk-on metal grates etc.). Such companies are ISG (Industrial Service Group Inc.),
Williams Industrial Services Group LLC, EMCOR Group.
Concluding, from the results of this research it is unveiled that although there are firms that provide
a wide variety of services, the global market needs more integration. From all the examined
companies, Williams Industrial Services Group LLC seems to provide the most services, API
Services and ISG (Industrial Service Group) focus on specific sectors, Tetra Pak combines
manufacturing and servitizing in a very intriguing way. Also, of great importance is that Williams
Industrial Services Group LLC specializes also in energy production plants. They provide services
specifically designed for Nuclear, Fossil, and Hydropower Power Generating Facilities. In the list
provided below, a list of the most commonly offered PSS is presented. Namely, the following PSS
cover a wide variety of major industrial activities, including resources management, asset
management, production planning and scheduling, energy management and distribution etc..
• Skilled Craft Labor and Supervision
• Craft Labor Management
• Refueling Outage Support
• Capital Project Support
• Plant Modifications
• Work Planning & Scheduling
• Quality Assurance / Quality Control Programs
• Systems Modifications / Upgrades
• Security Screening / Background Investigation Services
• Tools & Equipment Management
• Reactor Head Maintenance
• Steam Generator Replacement Support
• Vessel and Heat Exchanger Replacement
• Condenser Tubing and Cleaning
• In processing, Security Clearances & Background Investigations
The above list is used as a reference to what extent their services expand.
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 23
Division of Design & Manufacturing
Leasing/renting equipment/machinery: If a new enterprise is to be established and the budget is
low, or if specific machinery or equipment is needed for a known time period, then there are
solutions to keep costs low while acquiring the needed equipment. Byline Financial Group, API
Services and CUSHMAN & WAKEFIELD are two companies that provide leasing/rental services.
Data acquisition and analysis: In global market, all companies are striving for maximum profit
and sales, while maintaining customers’ interest for their products. Thus, the need for market
analysis, finding new trends, defining challenges, and seizing opportunities, is obvious. Not only
that but a company also has to ensure its steps (e.g. expanding to new markets, launching new
products etc.). the solution to this problem is given by firms, such as Hoovers, Euromatic
International and CMIE (Industrial Analysis Service). These companies have databases (which are
updated on a regular basis) and upon customers’ request they collect all the needed data, analyze
them, and send back reports according with the customers’ needs.
Another point that should be taken into consideration is that ISG (Industrial Service Group Inc.)
provides services relative to plastics injection molding industries. In the official website it is stated
that the company provides precision cleaning of hot runners. They also service plastic processing
components (e.g. molds, dies, tooling).
2.8 DATA PROTECTION AND DATA ENCRYPTION
With the use of Cloud computing, a big amount of data is exchanged at any moment. Cloud
computing offers a variety of benefits to its users, while containing a factor or risk. Data being
transferred need to be secured and encrypted in order to prevent attacks and leaks from hackers.
In fact, although the technological advances in IT technologies shift to the use of Cloud services,
many organizations hesitate to migrate their data to Cloud due to security issues [52].
In order to cope with the rise of cybercrime, data encryption techniques are used. According to
Ayers [53], there are many reasons that dictate the use of encryption in all kind of data that are
being transferred. First of all, if any kind of data is being transferred, then it is considered as
vulnerable as it can be stolen during the transport. Second, in some cases of cybercrime, targeted
data are not to be stolen, but to be altered in a way to perform acts of fraud. Moreover, a serious
amount of data is considered as sensitive data. Sensitive data have to be encrypted in order to
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 24
Division of Design & Manufacturing
ensure users’ privacy and anonymity. Finally, a big amount of data is transferred through devices
(e.g. through mobile phones, computers etc.), thus creating a need for protection of these data, not
only while being stored but also while being transferred between devices. Following, is a table
with some of the most common security techniques used in server protection.
Table 3 List of common server protection techniques
A/I Security Measure
1 SSH Keys
2 Firewalls
3 VPNs and Private Networks
4 Public Key Infrastructure and SSL/TLS encryption
5 Service Auditing
6 File Auditing and Intrusion Detection Systems
7 Isolated Execution Environments
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 25
Division of Design & Manufacturing
CHAPTER 3. PROPOSED METHODOLOGY AND SYSTEM ARCHITECTURE
The tool proposed in this document is targeted for used with Android tablet or other similar
devices. Thus, a considerable amount of development has been done in order to utilize as much as
possible of the Android functionalities. Android platform is popular among worldwide technology
users, so it is wise that used against other platforms. The tool alone is not enough for the proposed
method. At least a computer has to be utilized to backup tool’s operations as a Cloud server. The
server side is developed in parallel to the application proposed, in order to communicate with the
application. The first section of this chapter, focuses on the presentation of the PSS model adopted,
based on the requirements for the developed tool. In the second section, the system architecture is
explained in detailed and the remainder of the chapter, focuses on the discussion of the developed
tool functionalities.
3.1 PSS MODEL
The developed system composes a product-oriented PSS solution that supports the product with
the important service of maintenance throughout its lifecycle. The PSS proposed can be acquired
either when buying a new mold or as an aftersales service, in order to support older equipment and
equipment sold by third party companies. The function of this PSS is dependent to the product
from its design phase, where the engineers make the 3D CAD components and assemblies of the
equipment. The CAD files are needed, in order to be converted to suitable 3D geometries based
on which the creation of 3D and AR scenes is done. Moreover, with the use of smart algorithm,
running on the CAD program, a datasheet with the assembly and disassembly steps can be
automatically generated. The proposed framework includes all the needed methods that combine
the 3D geometries and the steps created in order to create a series of 3D/AR scenes that guide
workers through the steps of assembly and disassembly of the equipment. At this point, it is
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 26
Division of Design & Manufacturing
stressed out that the algorithm can run in real-time (e.g. in case of new equipment) or near real-
time (in case of previously maintained equipment) where the scenes have already been created.
The procedure followed by the framework is depicted in Figure 8. In the figure, the components
of the flowchart are divided into steps, according to the actions taken during the maintenance
procedure of a machine, using the proposed approach.
Figure 11 PSS system flowchart
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 27
Division of Design & Manufacturing
More specifically, the first step, which is presented in Figure 9, integrates all the needed
steps/actions that take place, during the inspection phase of the machine, starting from the premises
of the client when they request maintenance of their equipment by sending a report to the
engineering department of the PSS supplier.
Figure 12 First step in the sequence of maintenance
Continuing in the procedure of maintenance, is the report exchange phase between the engineer
and the client, where the time and cost estimation is performed. This is an important step, as it is
the time when the maintenance/repair tasks are discussed and agreed. When, the tasks are agreed
from both sides then the final time and cost estimation are reported to the customer. The
importance of this step is boosted by the fact, that the proposed approach, based on historical data
(which are constantly updated on the Cloud database), helps the maintenance engineer, produce
more accurate estimations. The following figure (Figure 10), presents the aforementioned
components.
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 28
Division of Design & Manufacturing
Figure 13 Second step in the sequence of maintenance
The third step in the maintenance procedure is the actual undertaking of the maintenance tasks
agreed in the previous step. This set of tasks has not been affected from the implementation of the
proposed approach. The only advancement is the task assignment to each technician. In Figure 11,
the included components of the flowchart are presented.
Figure 14 Third step in the sequence of maintenance
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 29
Division of Design & Manufacturing
Following the repair/maintenance of the mold is the validation of the jobs done (). During this step,
the technicians, test the machine based on the machine and production line specifications, which
are retrieved from the Cloud database. If the validation procedure is completed successfully, then
the maintenance can be considered successful, thus the machine is sent back to the client’s
premises.
Figure 15 Final step in the sequence of maintenance
3.2 SYSTEM ARCHITECTURE
The developed framework includes a series of methods that facilitate the development of AR
assembly/disassembly instructions and inspection of equipment based on predictions from a
formerly created knowledge database and data from sensors, integrated to the mold (e.g. cycle
counters). During maintenance, three different parties participate: (i) the customer, which owns
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 30
Division of Design & Manufacturing
the equipment and uses the service, (ii) the engineer who has the expertise and is responsible for
scheduling and overseeing maintenance and finally (iii) the technicians, who are responsible for
the maintenance tasks. As it is presented in Table 5, in the designed application there are 3 major
user-groups and one group with administrative rights. The division of the users into the presented
groups is done so that each user can view only the data required to accomplish their tasks, whilst
maintaining their cognitive level and information feed to a minimum level. The provision of
administrator users has been given so that changes can be made from authorized personnel from
any point in the world just by accessing server’s domain through the corresponding graphical user
interface (GUI).
Table 4 Involved actors and their rights
Actor Responsibility
Client Sends and receives information about the mold characteristics and
maintenance tasks.
Engineer Supervises the workers, send reports, inspects annotations, and creates the
Bill of Processes, makes the fault diagnosis, has access to the AR tool and the
database, builds the scenes for the AR/3D tool.
Technician Makes the fault diagnosis, identifies the code of the mold, makes annotations,
has access to the AR tool and the database.
Admin Has access to all interfaces and has rights to modify the database.
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 31
Division of Design & Manufacturing
Figure 16 System architecture - the numbers present the sequence of interactions
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 32
Division of Design & Manufacturing
3.3 DATA FORMALIZATION
The Unified Modeling Language (UML) is used in order to model the proposed system. UML
stands for a standardized model to describe an object-oriented programming approach. Hence, it
was used in order to design the system architecture for the proposed methodology in an
understandable and structured way. UML consists of classes. Each class contains several data and
objects. While this is not enough to fully describe the proposed framework, UML, also provides
all the required tools, in order to describe dependencies and interactions between classes. Taking
into consideration all of the above, the author, created a UML diagram of the proposed framework
in the very beginning of the development described in chapters 2 through 4.
In Figure 51, the final version of the UML diagram, for the proposed framework is presented. In
order to conclude to this diagram, initially, the test case requirements, were discussed. Following,
the sets of data to be involved in the framework were determined. Following the determination of
the datasets, is their categorization to the corresponding classes. Having the basic infrastructure
ready, the interactions between the classes are modeled.
At this point, it is stressed out that every step in the design of the UML diagram is posed a challenge
for the author. Each dataset had to be determined carefully, in order to define the correct type of
data either to be expected or to be exported from each class so that confusion is avoided. It is
important, also, that these steps are designed carefully, as the UML diagram serves as the guide
for the actual development of the tool. However, during the development of the tool, as mentioned
in chapter 4, the developed tool, was reviewed in webinars and workshop meetings. These
intermediate evaluation procedures assisted the author to refine the UML presented in Figure 51,
and moreover several other aspects of the test case emerged, which led to the implementation of
more features to the tool, thus creating new dependencies between the initial classes.
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 33
Division of Design & Manufacturing
Figure 17 UML Class Diagram supporting the data modelling of the proposed framework
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 34
Division of Design & Manufacturing
3.4 APPLICATION FUNCTIONALITIES
The GUIs shown in this section are developed in Unity 3D™. Unity 3D™ is a powerful game
engine, which supports augmented reality application development with the use of the Vuforia
add-on. The CAD files used, are requested from the equipment manufacturer directly in a universal
form (such as stl, step, iges) which are then imported in Blender, a free and open source 3D creation
suite, in order to convert them to a suitable filetype for the Android application (such as dae,
Collada or obj files).
The colors used in this application where chosen based on the colors Microsoft uses. The main
criteria for color choosing are:
Vividness: Textual and 3D content must be clearly visible under all circumstances, especially
while working on harsh environments – such as a machinery shop. For this purpose, a black
background combined with green textual content is used.
Eye relaxing: this application is meant to be used for long hours; thus the user must not feel
anxiety, tiredness or any other feeling related to smart-device use and/or augmented reality use.
Figure 18 Introductory Graphical User Interface (GUI), MMT®
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 35
Division of Design & Manufacturing
Registering: Every user of the service must be registered in order to use it. It is a simple process,
with a minimum of 4 pieces of information needed:
• Username or email address.
• Personal code.
• The user group in which user belongs.
• Email address
When user fills all the information, the service automatically saves his data in a remote server. The
online server is used to handle all users’ data and recall them whenever needed. It must be noted
that no sensitive data are required for registration, thus users can feel free to use the tool without
risking getting stolen valuable personal information. Finally, when a new users registers to the
tool, an email is sent automatically from the tool, containing the credentials and user-group which
was selected during the registration phase.
Figure 19 GUI for the registration of new users in the platform, MMT®
Login: After initial registering, user can start using the service just by logging in using their
information. It must be noted, that after registration, user is redirected to the login page,
automatically. Then the service recognizes in which user-group the user belongs, so that the correct
features are loaded. Each user-group has access to specific features of the appication or different
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 36
Division of Design & Manufacturing
configurations of the same features (e.g. the AR tool is common for all users but client cannot use
AR as an assembly/disassembly manual).
Figure 20 GUI for the login of registered users, MMT®
Client’s Start Page: In Figure 19, the clients’ start page is presented. As it can been seen in the
above-mentioned figure, a wide variety of functionalities have also been developed for the
clients/customers, in an attempt to facilitate the data gathering from the production environment.
By extension, this will improve the completeness / integrity of the cloud repository, for future
reference.
Figure 21 Client's start page GUI, MMT®
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 37
Division of Design & Manufacturing
More specifically, from this GUI, clients can inspect their equipment in AR mode (health
monitoring, explained in later section), request design of a new PSS according to their needs. After
completing all the required data, the expert engineer, designs the PSS and informs the client.
Figure 22 PSS customizer interface, MMT®
Also, from this GUI the clients, can view a date picker applet (calendar) which is presented in
Figure 21.
Figure 23 Date picker interface, MMT®
When using this functionality, clients can request and schedule a maintenance program, taking into
consideration the availability of the maintenance-contractor’s production schedule. Using the
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 38
Division of Design & Manufacturing
screenshot button, users can take a picture of their equipment in High Definition. The screenshots
are automatically saved on the Cloud database, and then they can be sent supplementary to the
malfunction report, in order to describe the problem more adequately. Finally, in the customer
profile, each customer, can view and edit their personal data, as illustrated in Figure 22. It is
stressed that during the first login, i.e. following the user registration, the clients are redirected to
this GUI in order to complete their profile.
Figure 24 Customer information sheet, MMT®
These data are edited from each customer and saved to the Cloud database, so that the maintenance
contractor can keep track of their customer list. In the corresponding following section, the
maintenance engineer, can view a list of all the customers serving.
Finally, the client’s start page, has also available a “Report” button, which leads the corresponding
customer to the report creator GUI. The report creator is presented later on this chapter, as it is
common for all the user-groups. What is worth noticing, is that whenever a customer, submits a
new report to the system, the users, that are registered as “engineers” to the Cloud database, are
notified via email, to the email address, they used for registration.
Engineer’s Start Page:
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 39
Division of Design & Manufacturing
Figure 25 Engineer's start page, MMT®
With the mold inspection button, the AR mode opens, and the engineer can inspect the arrived
piece of equipment in AR mode.
With the download file button, the engineer, gains access at the file repository of the Cloud
database. The download interface opens (Figure 25), and the user can select through the available
files. Also, the user has to select a folder (from the provided list of folders), in order to save the
corresponding file. When the download is complete, user gets notified from the tool that the action
was completed successfully.
Figure 26 Download file interface, MMT®
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 40
Division of Design & Manufacturing
With the “BOP” button, the engineer is directed to the BOP creation GUI, where a set of dropdown
lists is presented. Using these dropdown lists, the engineer, selects through the available list of
machines and the corresponding tasks. After finishing the composition of the BOP list, the
engineer can export the list to the Cloud so that the tasks can be assigned to the workers. At the
right lower part of the GUI, a progress bar can be viewed (Figure 25), which informs the user for
the progress of the undergoing task.
Figure 27 BOP creation GUI, MMT®
At this point it must be noted that the BOP can be exported as an XML file. The XML file is
structured as presented in Figure 26.
Figure 28 XML file structure for the generation of BOP, and generation of AR instructions
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 41
Division of Design & Manufacturing
Therefore, it becomes obvious from Figure 26, that elements are used in the XML structure. The
elements were used in order to add more flexibility to the produced XML file.
In Figure 27, an example XML exported directly from the tool is presented, featuring a set of three
tasks, along with their specifications, such as the sequence number, the task name, the task
description and the product ID.
Figure 29 Example XML file based on the structure presented previously
The engineer, however, from the BOP GUI, can use another functionality, hitting the “History”
button. In the history GUI, the engineer, inserts the desired date, the desired mold model, or both,
and the tool makes an inquiry on the Cloud database, in order to fetch, if any available, historical
data.
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 42
Division of Design & Manufacturing
Returning to the start page of engineers there is another crucial functionality, described as the
alarming tool. Using that tool, the engineer, gains access to all available machines, and views all
the available info, as well the expected time to arrive for maintenance. This functionality is useful,
as it assists the engineering department of the maintenance contractor, to schedule their production
line, based on the machines expected to arrive. In order to inform the maintenance engineer about
the incoming mold, the corresponding listing in the mold list is colored red, indicating an alarm.
In Figure 31 the mold-list GUI is presented. Furthermore, this functionality, exports a variable, the
availability of the maintenance contractor, which is used in the date picker functionality, presented
previously, in which the client, views the available dates for maintenance.
Figure 30 Mold-list GUI
Finally, using the “Shop Inspection” functionality, the corresponding GUI opens, enabling the
engineer to get an overview of the availability of the machines at the shop floor, without the need
to get to the production line, physically. For the machine availability, three colors are used. Red is
used for machines NOT available; green is used for available machines and finally yellow is used
for machines that will be soon available. This functionality, aims to assist the engineer, organize
the maintenance contractor’s production line more efficiently. In Figure 30 the “Shop Inspection”
GUI is presented.
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 43
Division of Design & Manufacturing
Figure 31 Shop Inspection GUI, MMT®
Technician’s Start Page: In Figure 31, the starting GUI for the experienced technicians at the
shopfloor is presented. What is worth noticing, is the access to two modes of AR, on the one hand
being the inspection mode and on the other hand being the assembly/disassembly mode. Moreover,
the technician can view a 3D presentation of the mold he/she is working on. Hitting each button
from the ones listed above, leads the technician to the corresponding GUI. These GUIs are
presented in the remainder of this section.
Figure 32 Technician's Start Page, MMT®
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 44
Division of Design & Manufacturing
3D View interface: A GUI similar to the AR GUI. User sees a 3D representation of the mold. The
buttons add functionality to this interface. After hitting the “Menu” button at the upper right corner,
the user is able to select through different functionalities, see the following table.
Table 5 List of additional functionalities
A/I Functionality name
1 Exploded view
2 Annotation
3 Step selection
4 Object rotation
While working on that interface, user can select any component he wants to interact with or even
select the whole assembly. In order to clarify if a component is selected, when the user taps on it
(thus selecting it) it becomes green and returns to its default color (white) when deselected.
Figure 33 3D presentation interface, MMT®
Exploded View: Exploded view offers the user, the opportunity to view all the components of a
mold moved away from each other. Hidden objects, such as internal components and
subassemblies are now clearly viewed. This functionality is based on the assembly instructions
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 45
Division of Design & Manufacturing
created during design phase of the mold. The service’s algorithm analyses assembly steps and
component tier and accordingly translates them in a way that everything is visible and interactable
through the smart-device touchscreen. This functionality is accompanied by a slider, which
translates its movement to distance between the parts. When slider is returned to its initial position,
the components are put to their positions as when assembled.
Figure 34 Explode functionality in 3D presentation, MMT®
Annotation Button: When selected, user has the ability to select a specific component or even
select the whole assembly and make useful annotations. These annotations may contain
information about maintenance tasks performed or information about any unusual measurements
taken throughout the inspection phase. The annotations are saved to the server so that when the
next person in the production line/machine shop gets the components of interest, using the tool can
retrieve from the server all the annotations made so far. The tool has been developed, so that the
user can easily select through the components of the machine. In order to select the component of
interest, user has to tap it. The selected component, then turns green, in order to notify the user of
his selection. In order to deselect any component a double tap is required. By default, if the user
has not selected any of the available components, and yet he submits a new annotation, the
annotation is registered to the whole assembly. In Figure 33, the selected component can be seen,
as well as the input field for the annotation is visible on the upper left corner of the GUI.
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 46
Division of Design & Manufacturing
Figure 35 Annotation functionality in 3D presentation, MMT®
The same sequence if followed when the user wants to retrieve annotations made in the past. If
there are any available annotations saved on the Cloud database, then they will be displayed on
the user’s GUI. In Figure 34, once again the selected component can be recognized. What is more
on the top center portion of the GUI, a previously made annotation on the selected component is
presented (circled component).
Figure 36 Annotation on 3D component, MMT®
In order to boost functionality of this GUI, several other components have been implemented, such
as the “Explode” button, which is an option to explode the machine’s components so that the inner,
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 47
Division of Design & Manufacturing
yet not visible, parts can be manipulated. What is more, the 3D model can be manipulated, in terms
of rotation and scale, by using the “Rotate” button and “pinch” gesture (common gesture in
Android devices) accordingly.
Select Step Button: During the designing phase and after completing the whole assembly of the
equipment, the engineer using an add-on attached to the CAD program can produce a sheet with
the assembly/disassembly steps of the equipment automatically. This sheet is then used by the
presented tool in order to import the sequence of all assembly/disassembly steps that need to be
followed. Then a dropdown menu is created automatically. The available options of the
beforementioned dropdown menu are created automatically from the tool after reading the sheet.
What is left, is the user to select the step which he is interested in to view guidance. After step
selection, user guidance begins, either in 3D mode or AR mode, using animations wherever
possible accompanied by 3D textual content. User at any point of the assembly/disassembly
sequence can pause, stop, and use any of the described functionalities provided with the tool.
Report tool: Report is similar to the annotation tool, but they serve different purposes. Reports can
be created from two user-groups, the clients, and the engineers. The report creator GUI is presented
in Figure 35.
Figure 37 GUI for Report creation & submission to the Cloud platform, MMT®
Clients use this functionality in order to write maintenance requests to the equipment manufacturer
and to report any unusual behavior from the machinery. Engineers use this functionality, in order
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 48
Division of Design & Manufacturing
to write time and cost quotations. In the report GUI, a text editor appears where the users can write
the content they wish. Then there is the “Submit” button which, when pressed, sends the report to
the server. The file sent to the server can be saved in multiple types. The formal type is exported
as PDF, the file aimed for internal use, is saved as TXT file and finally, in the server there is a
table where all the components of the report are categorized and saved separately. The report
repository can be viewed in Figure 36.
Figure 38 Reports, saved on the Cloud FTP server
Rotate Button: As the name implies, hitting that button, the user can rotate the whole mold
assembly on his screen in order to view hidden spots, or help view from a more convenient point
the mold. At this point it must be clarified that once hitting “Rotate” button the user can rotate 3D
objects. Hitting again the button and the functionality is disabled.
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 49
Division of Design & Manufacturing
Figure 39 AR visualization of the exploded assembly
Mold Inspection: As discussed earlier, it is crucial to ensure that an injection mold works under
the specified parameters, in terms of pressure, vibration, and temperature. The service provides a
tool so that a customer who has retrofitted the necessary sensors to their mold can view in real
time the condition of the components of interest. The working principle of this functionality is
simple. The data collected from the individual sensors are stored in suitable databases. The service
has access to this data, so that it can compare them to the ideal working values. For this purpose,
it is supposed that every measured variable has a min and max value. Thus, a working range is
created. If the measured value is somewhere between that range, then the component is “OK” and
the components are colored in green color. If the measured value lies on either of the verges of
the working range, then it might require client’s attention. Thus, the component(s) of interest are
colored in yellow color. Finally, if the measured value falls out of the working range, that means
immediate action is required, so the component(s) are colored in red color.
Finally, the components of interest are augmented on the tangible machine, in true scale and shown
on the user’s screen. The user can take a snapshot, using the corresponding functionality, of the
faulty component and save it on Cloud. When the machine is sent to the maintenance contractors’
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 50
Division of Design & Manufacturing
premises for repair/maintenance, the snapshot will be retrieved from the Cloud, in order to assess
the machine’s condition. The measured value is saved on a report sheet.
Alternatively, if the client has not yet installed sensors on their machine, data can be estimated
from the working cycles, based on the measuring from the already installed cycle meter.
Figure 40 AR mold projection, health monitoring
Server side: As discussed in the introduction of this section, the tool is supported by a server. The
development of these two components (application and server) is done in parallel. Actually, server
acts as the backbone of the tool presented. These two components are dependent to each other.
Server holds a database in which data sets from the tool are saved. Also, the server supports FTP
suitable for file exchange between users. The functionality lying here, is the creation of a cloud-
based CAD exchange, between the client, the manufacturer and the company that undertakes
maintenance tasks, in case it is different from manufacturer. In case of customizer, client can
request a new product or modify an existing one. In either case the tool supports FTP transfer
protocol so that the client provides the required CAD files. Also, in customization, another feature
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 51
Division of Design & Manufacturing
is the addition of sensors in points/components of interest so that the health of the product can be
monitored in real-time. In this case the CAD files, are used in order to produce AR scenes
presenting the position in which the selected sensor(s) must be placed on the product.
Server is also needed for data handling. Data, acquired from sensors or extracted as input from the
users, are saved automatically to the server. Timing and cost estimation can be a risky task,
especially if the estimation is about new tasks. In order to encounter that problem, the proposed
method, consists of a table which holds data about all the tasks performed during maintenance of
equipment and time and cost. With this approach, when the customer requests a maintenance plan,
the assigned engineer can accurately with the help of the tool to estimate time and cost for each
maintenance task according to the data-tables held on the server. Moreover, the data acquired from
the sensors of a machine (in case a machine has integrated sensors), are used from the application,
in order to inform the customer (the end user of the equipment), at all times about the health of his
equipment. The way, machinery health, is monitored is described in previous section.
Figure 41 Cloud database components
Furthermore, in order to accomplish communication between the tool and the Cloud, a set of
special scripts were written. The language used for that scripts, is PHP. The aforementioned
programming language was selected among others (e.g. HTML), as it is free, it is user-friendly, it
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 52
Division of Design & Manufacturing
is compatible with HTML and there is plenty support on the internet. Due to the needs of the tool
for constant communication with the Cloud database, several PHP scripts were written and
uploaded to the Cloud database. Each time the tool needs to communicate with the Cloud, the local
script (running in the background of the tool) calls the corresponding script and the communication
is established. Thus, the tool requires constant access to the worldwide web. For that purpose, each
time the application starts, checks internet connection, and notifies users accordingly. In Figure
41, a typical example of a PHP script developed is presented.
Figure 42 Typical paradigm of PHP script
Administrator user-group: A special user-group, has also been implemented to the developed
tool, in order to offer extra functionality to the users, while ensuring the application’s good
operation. For that purpose, the administrator has access to all the application’s interfaces, as
presented in Figure 42.
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 53
Division of Design & Manufacturing
Figure 43 Administrator interface, MMT®
Additionally, the administrator’s GUI has been enriched with the provision of administrative tools
(Figure 42). These tools are used to process the Cloud database (e.g. create, edit, and delete tables
in the Cloud database).
Figure 44 Administrative tools interface, MMT®
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 54
Division of Design & Manufacturing
CHAPTER 4. INDUSTRIAL TEST CASE
In order for the utilization of the proposed platform and its functionalities to be demonstrated, a
case study is presented below.
A mold manufacturing company, responsible for creating and maintaining molds, is selected for
the case. The mold company selected for the test case, is a high-end SME located in Greece, with
a total count 50 people personnel, manufacturing state-of-the-art, highly accurate Engineered-to-
Order (ETO) tools and equipment. The mold company is specialized in producing highly accurate
precision parts, progressive cutting and forming dies, injection molds and undertakes demanding
product development and industrialization projects, in collaboration with OEMs.
The selection of this manufacturer, as the test case to implement the proposed framework poses a
great challenge. That is, because of the nature of the tasks this company has to accomplish. The
company, as mentioned before specializes in the manufacturing of ETO IM, thus there is low to
no repeatability in the maintenance tasks of each IM. Moreover, taking into consideration the
delicacy of the components consisting an IM, its extended life cycle is highly dependable on the
maintenance and repair services. Currently, the company offers such services only in the context
of the after-sales plan, which creates an opportunity for the enterprise to adopt servitization in its
business model.
Regarding other aspects of the IM maintenance, it is often impossible to plan maintenance
operations. Thus, the design and development of a product-based PSS oriented in preventative
maintenance, could possibly offer a robust solution for the PSS-supplier, who also gains their
customers’ loyalty. In the remainder of this chapter, a possible solution to this problem is proposed
with the use of sensors, installed on the IM. In conjunction with the proposed framework from
Chapter 1, the provision of an all-around, customizable PSS solution is presented. The customer
has the opportunity, when buying a new mold to buy also, a PSS which will automatically extend
his IM life cycle, thus creating more added value for the physical assets of their company.
However, the provision of such solutions, is not available, only for new customers. The proposed
PSS is designed, so that with the addition of specific features, and the acquirement of the bare
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 55
Division of Design & Manufacturing
minimum equipment (which may already exist in the company), customer can benefit even though
they do not purchase new equipment.
The molds for maintenance may be produced by the company itself or from other manufacturers.
Nowadays, the maintenance in this industry is done non-automatically. The customer sends the
mold in the company’s premises and workers after thorough examination of the mold, identify the
jobs that need to be done in order to restore the mold. A report is sent back to the customer and if
he approves the maintenance activities, he sends a document with some measurements and
characteristics of the molds so as to proceed with the maintenance. The maintenance tasks start,
and the workers are responsible for them. Then the workers monitor the duration of the
maintenance and send it to a supervisor engineer to fill in some documents for the duration of the
tasks.
The mold manufacturer receives the mold that needs to be maintained and through a smart device
the worker inspects the parts. The maintenance history is also checked so as to extract information
about the time and cost of previous maintenance activities. During mold maintenance, the service
provides a smart tool able to monitor the maintenance and another AR tool able to simulate the
assembly and disassembly of the mold. Although there are tools fully automated in the service, the
role of the user is not neglected. The case study consists of three actors shown in the Table 4.
Figure 45 The mold manufacturing company
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 56
Division of Design & Manufacturing
4.1 INJECTION MOLD OVERVIEW
The process of injection molding is fairly simple, yet there are areas, which if are not inspected
can destroy the end product and the production flow. Injection molding consists of three tasks, (1)
melting the media to be injected (plastic), (2) injection of the melted plastic and (3) ejecting the
finished product/component [54].
Injection molding is a repetitive process. Initially the plastic is heated up to its melting temperature,
so that moisture escapes and plastic can be easily injected. Then the hot plastic fluid is injected
through the injection unit to the mold cavities.
Molds consist of plates (stationary and moving) which when are sealed together, form the cavities
in which the hot plastic is injected. If production rate demands the injection of multiple pieces at
a time, then plates have more than one cavities. Cavity shapes may differ to each other. In order
for the hot plastic to run through all cavities, channels are grooved on the plates. These channels
are called runners (or hot runners). All the above components can be viewed in the Figure 44,
which presents a cut-out of a typical injection mold (IM).
Figure 46 Injection mold cut-out
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 57
Division of Design & Manufacturing
During and after injection, the plastic stays pressurized in the cavities until it solidifies. After
some seconds, the product has cooled to a point where it can be ejected from the mold. Injection
mold cycle has completed and its ready to start over. The duration of each cycle is lasting some
seconds, which depends from the product’s characteristics as well technical specifications of the
mold machine itself. In Figure 46, a complete injection cycle is presented with the corresponding
times of each individual task. The maximum time of a complete injection cycle, lasts up to 24
seconds, depending on the characteristics of the final product.
Figure 47 Injection mold cycle with the corresponding times
There are several points throughout the injection molding process where changes in an asset’s
temperature, pressure, or humidity can compromise product quality. If left unchecked, various
imperfections in moldings and in production line can happen. A list with with the most common
faults is provided in Table 6.
Table 6 List of common faults for injection molds
A/I Fault Name / Description
1 Part deformity
2 Stress cracks
3 Extra flashing
4 Excess scrap
5 Increased recall costs
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 58
Division of Design & Manufacturing
A/I Fault Name / Description
6 Trapped air/moisture
7 Shrinkage
8 Voids in moldings
9 Overall poor end result
10 Unmet specifications, like especially tight tolerances
In Table 6, a list with the common failures of IM is presented. However, for the development
of the proposed PSS, that information is not enough. To encounter that probem Figure 46, is used
instead, as it lists the failures and the corresponding actions needed, to properly repair and
maintenance an IM.
Figure 48 Injection mold common failures
In order to prevent the beforementioned damages and ensure top quality of the molded
products, sensors can be installed on key points of the injection mold and monitor vital
components’ health, a crucial step for preventive maintenance. These sensors are:
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 59
Division of Design & Manufacturing
Accelerometers: Accelerometers are used for vibration measurement (excessive vibration is
most likely to cause of wear and tear, as well as of damages to the mold).
Thermocouples: Thermocouples, are used for mold temperature monitoring.
Cavity Pressure Sensors: Cavity pressure sensors, are used for analyzing the quality of the
injection process.
Consistent and accurate condition monitoring is critical to ensure the molding process is not
costing the company thousands annually in additional operational costs.
Temperature monitoring: If equipment is running too hot or too cold, the two seconds it takes
for the initial mold process can be enough to jeopardize product quality. From the resin storage
area to the mold itself, there are varying temperatures that must be kept consistent. (e.g. clogged
water channels in the injection mold will not show an increase in water temperature, but cavity
temps will rise, hindering the cooling process and leading to product deformations)
Pressure monitoring: From the injection screw that pushes material into the mold to the clamp
that holds the mold shut, insufficient pressure can let air and moisture seep into the material,
leading to batch defects:
Short shot: A void of material in a portion of the part due to insufficient pressure.
Flash: Excess pressure forces mold open just enough to allow material seepage, resulting in
defective parts and potentially damaging the mold.
Dimensional variation: Packed mold is fully compressed, but cavity pressure changes at a
critical point.
Warp: Variations in packing pressure (as well as cooling rate/time) that leads to random
distortions in part. Maintaining proper working pressure is a vital part of the injection process.
Humidity monitoring: Plastic resin is subject to changes in ambient humidity conditions in
storage containers, as well as the process lines. Too much humidity can add moisture to the resin,
changing its properties and preventing it from molding like it’s supposed to.
However, measuring ambient humidity is not enough. Humidity in the plastic itself, as well as
the moisture content of the air being blown from resin dryers has to be monitored in order to avoid
the creation of voids within the internal structure of the molded component/product.
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 60
Division of Design & Manufacturing
4.2 CASE STUDY DESCRIPTION
The first steps of the mold maintenance service are the same with the non-automated version
of the maintenance. Customer sends the mold and the technician in mold manufacturer premises
identifies the code of the mold that needs to be maintained by using a smart device and an AR tool.
The smart device will be a tablet and the scanning is realized with the camera. The identification
is based on a stamp that the mold has on it and by using this tool it is also possible to explode the
parts of the mold, see inside and retrieve information of all the mold components (ID, Name,
condition etc.). In Chapter 4, which deals with the future development, alternative solutions for
the tracking are presented. For the visualization of the mold in a smart device the CAD files are
needed. These files are provided by the customer or the mold manufacturing company retrieve
them from a database in case that he is the manufacturer. The service handles the communication
between the customer and the engineer and therefore the CAD files are provided on time and only
when needed. The communication feature of the service has three different dimensions, the
receiving of the CAD file, the mold characteristics (measurements, and production cycle
specifications) that the customer sends for the fault analysis and finally the report is sent to the
customer. The engineer inspects the mold together with the workers so as to conduct the fault
diagnosis. To this end, measurements are made in the mold and they are compared to the
measurements that the customer sent. In case of deviations, the engineer together with the worker
come to a conclusion related to the problems detected in the mold. By the end of the inspection
phase, the engineering department, composes the BOP and BOM lists required for the mold to be
repaired. Along with those two lists, the engineering department also composes a quotation which
reports the time and cost estimation for the malfunctions found or reported on the mold. By using
the mold maintenance tool, mold manufacturing company aims to create a database with mold
characteristics for every customer. In the current maintenance processes, each time that a
maintenance needs to take place the industry asks for additional information from the customer.
The existence of a database will support time consuming processes and the communication
between the industry and the client. A file with all the customers of the manufacturing company is
also already stored in the database. In case that the mold has been produced in the company, worker
gains access to the Bill of Processes and Bill of Materials of the mold. On the other hand, if
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 61
Division of Design & Manufacturing
maintenance has been already done in the company in the past, information about the cost and time
are retrieved from the database. An estimation is made for both of these values and is integrated
in the report that the engineer sends to the customer. When the mold is delivered for the first time
in the industry, the cost and time estimation is made by the engineer. As long as the customer
approves the report that the industrial partner sent, annotations should be made by the worker based
on the 3D model visualized by the AR tool. These annotations should be relevant to the
maintenance activities that are going to take place. In the mold manufacturing company, may be
more than one workers that are able to make annotations. Every worker has a smart device and is
able to read the annotations of the others and add new ones. This is also useful during the
maintenance tasks. The responsible workers see the comments of the others so as the processes to
be done properly. Each annotation may refer to a different component of the mold and they should
be legible and in a structured way. The engineer checks the annotations made and creates the Bill
of Processes (BOP) of the maintenance. The BOP consists of a list with specific maintenance tasks,
the corresponding machine, and the related worker. This will make the maintenance tasks to be
easily transformed in real tasks in the mold manufacturing company. During the maintenance
phase in the company, the mold maintenance tool records the real processes. The time used for the
tasks is stored in the database so as to be used in the future. One of the innovations offered is that
the worker is able to pause the time if a machine breaks down or something else occurs. Therefore,
the real maintenance time is monitored and not just the starting and end time of the processes. The
AR tool that the service provides is able to visualize the mold, or the component of the mold that
need to be maintained. The position of this component in the mold is also visualized. Before and
after the mold maintenance disassembly and assembly activities take place respectively.
Instructions about these activities are also generated by the AR tool. The order of the parts, the
tools used for the assembly/disassembly are information provided by the tool. The tools refer to
simple tools such as screwdrivers or pincers in order to be easier for the worker to proceed with
these processes.
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 62
Division of Design & Manufacturing
Figure 49 Application of the developed framework
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 63
Division of Design & Manufacturing
CHAPTER 5. RESULTS – KEY PERFORMANCE INDICATORS
5.1 KEY PERFORMANCE INDICATORS
Key Performance Indicators could be considered as any kind of metric to quantitatively measure
the performance of a system or the processes within an organization [55].
High repetitiveness and the generation of vast amount of data characterize manufacturing
activities [56]. Manufacturing systems support the production products. Thus, the focus is on
meeting customer requirements. Through the stage of manufacturing, feedback in the form of data
is gathered. These data, if processed correctly can produce physical assets’ results. Manufacturing
performance can be directly linked to quality, availability, customer service, operating costs,
safety, and environmental integrity [57]. Maintenance and reliability business metrics provide a
clean indication of compliance to the maintenance business process.
5.2 TEST CASE CURRENT SITUATION DESCRIPTION
The mold company selected as the test case is a SME located in Greece, focusing on manufacturing
and maintenance of injection molds. In the present, mold maintenance is not documented
adequately thus creating gaps in matters such as maintenance tracking, maintenance history of sold
molds, historical data about mold assemblies and the scheduling is performed by manually by the
mold maker engineering department. More specifically, if a customer requests mold maintenance,
the physical product is sent to company’s premises along with a document about the mold
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 64
Division of Design & Manufacturing
specification. Upon mold arrival, the company’s engineering department, receives the mold and in
conjunction with the technicians, the mold inspection is performed. Continuing, the engineering
department after collecting all the needed data, composes the BOP, along with a time and cost
estimation. The time and cost quotation is sent to the customer for approval. If customer’s approval
is granted, then the actual maintenance procedures take place. Finally, the corrected mold is tested
(if such an action is needed according to the company’s engineering dept. instructions) and is
returned back to the customer. Another identified gap is about following the maintenance plan
agreed during purchase contract. When the mold company sells a new mold to their customer, in
the contract a maintenance schedule is also dealt. Unfortunately, both mold company and the
customer cannot follow the dealt maintenance plan due to their production requirements. The
proposed framework has implemented an alarming tool, accessible from both stakeholders. The
system’s purpose is to alarm the customer that their mold needs to be corrected (preventive
maintenance) soon enough to reschedule their production. Finally, the last identified gap is about
unscheduled maintenance and lack of an alarming system for the mold company’s engineering
department in order to continuously keep track of all customers’ mold health. Currently, the
company, has no input from their customers, about mold health during their operation. Thus, mold
company’s production cannot be scheduled effectively, creating the possibility of mold stack in
their premises. The alarming system will serve as a tool to predict mold arrival for each customer
thus enabling company’s production planning team to prepare a long-term schedule for their
production.
5.3 EXPECTATIONS OF ADOPTING THE PROPOSED METHODOLOGY
In this section, the validation metrics for the usability testing of the tool are presented. The
validation should have two parts, the first one regarding the usability of the tools and the second
one regarding the impact on the business process under study. In order to facilitate the processes
of information gathering and validation, a questionnaire is setup regarding the pilot case.
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 65
Division of Design & Manufacturing
The validation scenario for the Mold Manufacturing Use Case focuses on the case of the
configuration, design, manufacturing, and usage of a Mold PSS. This scenario will also
demonstrate the value that will be provided to the mold user in later stages of the Mold Product
Service lifecycle by demonstrating the use of the provided PS for the maintenance of the mold.
The validation scenario includes all the steps for the configuration, design, and usage of a Mold
Product Service for the maintenance of the mold, later inside the Mold’s lifecycle.
This process is described in the following subsections in the form of a validation scenario that
is provided in a sequence of discrete validation steps.
The business process will be assessed not only subjectively based on the questionnaires but
also objectively by getting some measure. Towards that end firstly a qualitative validation through
questions to the experts (company’s engineers, workers, Customers) is presented and then specific
Validation metrics for this case are presented in Table 7. Table 7 KPIs used for the test case
KPI Definition As-Is
Value
To-Be
Value
How is it measured?
Time for
Inspection
Actual time (in days) from
the announcement of the
repair request until all
information is collected
and the team is ready to
inspect the mold.
2 days 1 day Date that the inspection takes
place - Date of the first
customer contact (E-mail or
phone).
Monitored by company’s
project manager.
Time for
inspection
documentation
Actual time (in working
hours) for the inspection
findings to be documented
1.5
hours
0.5
hours
The time spent to prepare the
hand-written service sheet
and digitally transfer the
notes in a quotation. Timed
by company’s project
manager.
Maintenance
History Time
Retrieval
Actual time (in working
hours) for examining past
maintenance cases.
3 hours 0.15
hours
The time spent to retrieve
information from past
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 66
Division of Design & Manufacturing
KPI Definition As-Is
Value
To-Be
Value
How is it measured?
maintenance cases. Timed by
company’s project manager.
On time
delivery time
Percentage of
maintenance jobs that
were delivered on time.
70% As-Is information available in
the legacy system (excel
based)
Total
Maintenance
Jobs
The total amount of
maintenance jobs in one
year.
64
(2016)
As-Is information available in
the legacy system (excel
based)
5.4 VALIDATION METHODOLOGY
The test case discussed above will undergo a validation process which is planned to be twofold.
The software usability, functionality and user friendliness will be evaluated with the use of
qualitative questionnaires. Additionally, the business impact of using the proposed solution is
going to be measured and quantified as objectively as possible. In order to do so, as there is no
current business process that can be directly compared to the PSS oriented process that the
proposed platform introduces to the company, the current work flow needs to be analyzed in steps
that can accommodate the use of objective metrics. Although this is mainly possible for test cases
3 and 4, there is currently no structured process of designing and managing PSSs to compare test
case 1 impact during project duration, nor an official PSS offering to the customer to compare with
test case 2. However, some metrics that can be used for assessing test cases 1 and 2 are defined,
yet the to-be figures cannot be estimated realistically.
The current service business process subject to validation is the mold maintenance process,
which is analyzed below in Table 8:
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 67
Division of Design & Manufacturing
Table 8 Mold maintenance process steps
Step 1. Request for repair
The customer identifies a problem with a mold and commences the process by communicating via e-mail or phone to the mold maker the identified issues, the mold serial number or description and sends the mold to the mold making company premises.
Step 2. Inspection
The mold is delivered to the mold making company. The mold serial number is cross checked for validation by the project manager, previous cases of maintenance are retrieved, and a hard copy service sheet is prepared, including the issues identified by the customer. A team of a project manager, an engineer and an expert mold technician are gathered to inspect the mold. Issues are identified and noted on the service sheet, solutions to these issues – basically machining operations - are also proposed and noted in detail. A rough estimation of cost and lead time is also being made.
Step 3. Quoting and confirmation
The project manager cross checks similar past cases of maintenance for consistency, discusses the specific case with the production planning team with respect to the current workload and the team estimates an actual delivery time. A formal quotation is then prepared and sent to the customer, which is confirmed in due time.
Step 4. Machining Operations
Following the confirmation from the customer, machining, procurement, and other necessary operations are executed and the mold is repaired/restored to its initial state.
Step 5. Test sample production and mold delivery
The repaired mold is put in operation in the mold making company premises, in order to validate the repair and that the tool runs at the desired – agreed condition. The mold is then prepared for pick up along with the test samples, and the customer is notified that the repair is completed.
In the following figure (Figure 51), a flowchart is illustrated, representing the sequence of
actions for the validation of the proposed framework. It must be noted that the procedure is divided
into three tiers. In the first tier the test case steps are described. In the second tier the developed
tool impact is presented through its main functionalities. Finally, in tier three the outcome, thus
the measured values are recorded. Based on these results the validation was performed.
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 68
Division of Design & Manufacturing
Figure 50 Workflow of the validation scenario of first prototype of the tool.
5.5 RESULTS
With the adoption of the proposed methodology derived from this thesis, the mold company,
although it is a SME, offers services to their clients, in order to maintain its competitiveness and
their customers’ loyalty. Moreover, any financial risk is eliminated due to the use of equipment
already existing in an industrial environment (tablets, smart phones etc.). However, if not the
required equipment is not existent its cost is really low to pose any kind of threat for a SME.
Also, in the test case presented in previous chapter, there was identified a gap of digital
organization for all the data required to perform maintenance. Thus, in many cases, the
maintenance procedure has been delayed in order for the maintenance contractor to acquire all the
needed data. With the implementation of the designed tool, the maintenance contractor, obtained
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 69
Division of Design & Manufacturing
access to these data more quickly and more easily. The results from the tool evaluation phase,
indicated a reduction of approximately 70% for the inspection documentation.
The proposed framework also serves as a communication tool between the maintenance
contractor and the client, thus enabling them to communicate more efficiently. Based on that, and
on the use of historical data, which are constantly saved on the Cloud database, the maintenance
contractor was able to accomplish more maintenance tasks, thus making their production line even
more efficient. The quantitative results gained from the evaluation phase indicated a 56% of
increase for the tasks successfully completed.
In Figure 52 the quantitative results are presented compared to the original values given by the
test-case company.
Figure 51 Tool evaluation results
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 70
Division of Design & Manufacturing
CHAPTER 6. CONCLUDING REMARKS AND OUTLOOK
The tool presented is constantly updated, so that bugs are encountered as well adding extra
functionalities which in turn provide better experience to the end user. Moreover, feedback from
actual users is needed so that weak spots of the tool can be easily spotted, and new ideas come to
the surface. Also, it must be taken in consideration that current technological progress in the field
of AR and generally in mixed reality environments, will provide in the near future capabilities that
will boost the performance of the proposed tool. The current set of ideas is discussed further in the
following section.
The first goal for AR developers is getting rid of trackers and image targets (markerless AR).
This task is currently feasible in some cases, but the results are not satisfactory enough due to
insufficient technological equipment and low feature recognition. Thus, further research and
development has to be conducted in order to accomplish a seamless connection of the virtual
content with the tangible objects. A convenient solution is the use of object recognition. Currently
based on the Microsoft Hololens sensing system the formation of a relatively accurate point cloud
is feasible. By extension, the augmentation/super imposition of the digital content on the real
environment is performed. Object recognition relies on the camera feedback, get a point cloud and
depending on what features it recognizes, a set of virtual contents is registered on the physical
environment.
The second goal is to support real time integration of CAD files to the tool. As FTP exchange
protocol is currently supported, meaning that CAD files can be exchanged between users while
communicating through the tool, is not enough, the next step is to convert them online and load
them without the need of a dedicated engineer to create the according scenes in Unity™ editor.
This task requires the integration of a file conversion tool, that will convert CAD filetypes to 3D
game engine filetypes (the kind of filetypes Unity™ and Android applications need to work with).
By accomplishing that task, instantly the application becomes lighter, as there is no longer needed
to package all the 3D geometries during releasing a version of the tool. The user would easily
select the model of equipment he wishes to work with and then the application should perform all
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 71
Division of Design & Manufacturing
the required actions to get the geometries from the server and present them to the user’s device.
All is needed is access to the server, implying the necessity of an internet connection or LAN
connection existence if the server is local.
As discussed in the previous paragraphs, client uses the customizer tool to order new pieces of
equipment or order changes to the existing equipment. For this purpose, the client is required to
provide at least the engineering drawings or complete 3D designs. In either case, a sandbox
functionality would provide much help in creating or modifying an existing piece of equipment in
real-time. For example, in the test case scenario, of the mold industry, described in the previous
section, it would be very helpful if the client could modify the shape of a mold’s cavity in such a
way that the produced molding could transform in the desired form, instead of providing a set of
drawings or even invest time to design the new cavity from the scratch.
Another goal is to use QR code system for equipment recognition. QR code system also known
a matrix barcode [59]. This system offers fast readability and great storage capacity, making it
suitable for industrial applications [60].
Finally, the application, the server and all of its components are controlled by an administrator.
Thus, the need for the suitable administrative tools rises. Currently, a set of administrative tools
has been developed, but over time as the application obtains more added value, expansion and
adaptation of the existing administrative tools might be required. It must be also taken in
consideration that a huge amount of raw data is accumulated over time, making necessary the
development of a series of sub-tools, for efficiently managing these data.
Finally, a topic of great importance is data security. The proposed framework has to manage a
big amount of data, which could be considered as sensitive. As a result, encryption techniques
have to be implemented. In the current state, of the developed framework, the data exchanged
between the smart device and the server, are encoded using the Base64 encoding system [61],
which is provides the least amount of protection.
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 72
Division of Design & Manufacturing
ACKNOWLEDGEMENT
This work has been partially supported by the H2020 EC funded project “An Integrated
Collaborative Platform for Managing the Product-Service Engineering Lifecycle – ICP4Life” (GA
No: 636862). The author would like to thank the industrial partner involved in this research work.
PUBLICATIONS RELEVANT TO THE THESIS
Part of the research work presented in this thesis has been published as a research paper in the
proceedings of the 16th IFAC Symposium on Information Control Problems in Manufacturing
(INCOM 2018), which was held in Bergamo, Italy, 11–13 June 2018. The conference proceedings
have been published by Elsevier as a special issue edited by Marco Macchi, László Monostori,
Roberto Pinto. The full citation of the published manuscript is the following:
• Mourtzis, D., Angelopoulos, J., & Boli, N. (2018). Maintenance assistance application of
Engineering to Order manufacturing equipment: A Product Service System (PSS)
approach. IFAC-PapersOnLine, 51(11), 217-222. DOI:
https://doi.org/10.1016/j.ifacol.2018.08.263
Below is a list of other research works published in scientific journals with high impact factor, and
proceedings of conferences indexed by Scopus.
• Mourtzis, D., Siatras, V., & Angelopoulos, J. (2020). Real-time remote maintenance
support based on augmented reality (AR). Applied Sciences, 10(5), 1855. DOI:
https://doi.org/10.3390/app10051855
• Mourtzis, D., Angelopoulos, J., & Panopoulos, N. (2020). Intelligent Predictive
Maintenance and Remote Monitoring Framework for Industrial Equipment based on
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 73
Division of Design & Manufacturing
Mixed Reality. Frontiers in Mechanical Engineering, 6, 99. DOI:
https://doi.org/10.3389/fmech.2020.578379
• Mourtzis, D., Angelopoulos, J., & Panopoulos, N. (2020). A framework for automatic
generation of augmented reality maintenance & repair instructions based on convolutional
neural networks. Procedia CIRP, 93, 977-982. DOI:
https://doi.org/10.1016/j.procir.2020.04.130
• Mourtzis, D., Angelopoulos, J., & Panopoulos, N. (2021). Collaborative manufacturing
design: a mixed reality and cloud-based framework for part design. Procedia CIRP, 100,
97-102. DOI: https://doi.org/10.1016/j.procir.2021.05.016
• Mourtzis, D., Angelopoulos, J., & Panopoulos, N. (2020). Recycling and retrofitting for
industrial equipment based on augmented reality. Procedia CIRP, 90, 606-610. DOI:
https://doi.org/10.1016/j.procir.2020.02.134
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 74
Division of Design & Manufacturing
REFERENCES
1 Mourtzis, D., & Doukas, M. (2014). The evolution of manufacturing systems: From
craftsmanship to the era of customisation, Design and Management of Lean Production
Systems, V. Modrak, P. Semanco. Pennsylvania: IGI Global. DOI: https://doi.org/10.4018/978-
1-4666-5039-8.ch001
2 G. Chryssolouris and F. Frederick, Manufacturing Systems : Theory and Practice Sprin ger.
1991.
3 Johnson, J. (2012). "The Master Key": L.Frank Baum envisions augmented reality glasses in
1901. Mote & Beam.
4 Wang, X., Ong, S. K., & Nee, A. Y. (2016). A comprehensive survey of augmented reality
assembly research. Advances in Manufacturing, 4(1), 1-22. DOI:
https://doi.org/10.1007/s40436-015-0131-4
5 Mourtzis, D., Zogopoulos, V., & Vlachou, E. (2017). Augmented reality application to support
remote maintenance as a service in the robotics industry. Procedia CIRP, 63, 46-51. DOI:
https://doi.org/10.1016/j.procir.2017.03.154
6 Mourtzis, D. (2020). Machine Tool 4.0 in the Era of Digital Manufacturing. 17th International
Multidisciplinary Modeling & Simulation Multiconference. DOI:
https://doi.org/10.46354/i3m.2020.emss.060
7 Romero, D., Bernus, P., Noran, O., Stahre, J. and FastBerglund, Å. (2016). The operator 4.0:
human cyber-physical systems & adaptive automation towards human-automation symbiosis
work systems. IFIP international conference on advances in production management systems.
Springer, Cham. DOI: https://doi.org/10.1007/978-3-319-51133-7_80
8 Liu, C. and Xu X. (2017). Cyber-physical Machine Tool – The Era of Machine Tool 4.0,
Procedia CIRP, 63, 70-75, DOI: https://doi.org/10.1016/j.procir.2017.03.078
9 Benotsmane, R., Kovács, G. and Dudás, L. (2019). Economic, Social Impacts and Operation of
Smart Factories in Industry 4.0 Focusing on Simulation and Artificial Intelligence of
Collaborating Robots. Social Sciences, 8(5), 143. DOI: https://doi.org/10.3390/socsci8050143
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 75
Division of Design & Manufacturing
10 Kollatsch, C., Schumann, M., Klimant, P., Wittstock, V., & Putz, M. (2014). Mobile augmented
reality based monitoring of assembly lines. Procedia Cirp, 23, 246-251. DOI:
https://doi.org/10.1016/j.procir.2014.10.100
11 Kiritsis, D., Bufardi, A., & Xirouchakis, P. (2003). Research issues on product lifecycle
management and information tracking using smart embedded systems. Advanced Engineering
Informatics, 17(3–4), 189–202. https://doi.org/10.1016/j.aei.2004.09.005
12 Mourtzis, D., Vlachou, E., Milas, N., & Xanthopoulos, N. (2016). A Cloud-based Approach for
Maintenance of Machine Tools and Equipment Based on Shop-floor Monitoring. Procedia
CIRP, 41, 655–660. https://doi.org/10.1016/j.procir.2015.12.069
13 Mourtzis D. (2020). Simulation in the design and operation of manufacturing systems: state of
the art and new trends, International Journal of Production Research, 58:7, 1927-1949, DOI:
https://doi/org/10.1080/00207543.2019.1636321
14 Zhu Z. and Xu X. (2020). User-centered information provision of Cyber-Physical Machine
Tools, Procedia CIRP, 93, 2020, 1546-1551, DOI: https://doi.org/10.1016/j.procir.2020.04.091
15 Mourtzis, D., Milas, N., Vlachou, E. and Liaromatis, J. (2018). Digital transformation of
structural steel Mourtzis D. | 429 manufacturing enabled by IoT-based monitoring and
knowledge reuse. International Conference on Control, Decision and Information Technologies
CoDIT’18. 295-301, DOI: https://doi.org/10.1109/CoDIT.2018.8394874
16 Liu, C. and Xu X. (2017). Cyber-physical Machine Tool – The Era of Machine Tool 4.0,
Procedia CIRP, 63, 70-75, DOI: https://doi.org/10.1016/j.procir.2017.03.078
17 BCG Analysis (2020). Embracing Industry 4.0 and Rediscovering Growth, Available online:
https://www.bcg.com/capabilities/operations/embracing-industry-4.0-rediscovering-
growth.aspx (Accessed 10/06/2021)
18 Azuma, R. T. (1997). A survey of augmented reality. Presence: teleoperators & virtual
environments, 6(4), 355-385.
19 Baines, T. S., Lightfoot, H. W., Evans, S., Neely, A., Greenough, R., Peppard, J., ... & Wilson,
H. (2007). State-of-the-art in product-service systems. Proceedings of the Institution of
Mechanical Engineers, Part B: journal of engineering manufacture, 221(10), 1543-1552.
20 Meier H., Roy R., & Seliger, G. (2010). Industrial Product-Service Systems-IPS2. CIRP Annals
- Manufacturing Technology, 607-627.
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 76
Division of Design & Manufacturing
21 Abernathy, M., Houchard, J., Puccetti, M., & Lambert, J. (1993). Debris Correlation Using the
Rockwell WorldView System. In Proceedings of 1993 Space Surveillance Workshop (Vol. 30,
pp. 189-195).
22 Lamata, P., Ali, W., Cano, A., Cornella, J., Declerck, J., Elle, O. J., ... & Gómez, E. J. (2010).
Augmented reality for minimally invasive surgery: overview and some recent advances.
23 Roberts, G. W., Evans, A., Dodson, A., Denby, B., Cooper, S., & Hollands, R. (2002, April).
The use of augmented reality, GPS and INS for subsurface data visualization. In FIG XXII
International Congress (Vol. 4, pp. 1-12).
24 Hammad, A., Garrett, Jr, J. H., & Karimi, H. A. (2002). Potential of mobile augmented reality
for infrastructure field tasks. In Applications of Advanced Technologies in Transportation
(2002) (pp. 425-432).
25 Thomas, B., Piekarski, W., & Gunther, B. (1999). Using augmented reality to visualise
architecture designs in an outdoor environment. International Journal of Design Computing
Special Issue on Design Computing on the Net (DCNet), 1(4.2).
26 Dunston, P., Wang, X., Billinghurst, M., & Hampson, B. (2003). Mixed reality benefits for
design perception. Nist Special Publication SP, 191-196.
27 Shen, J., Wu, Y., & Liu, H. (2001). Urban planning using augmented reality. Journal of urban
planning and development, 127(3), 118-125.
28 Behzadan, A. H., & Kamat, V. R. (2005, December). Visualization of construction graphics in
outdoor augmented reality. In Proceedings of the Winter Simulation Conference, 2005. (pp. 7-
pp). IEEE.
29 Oculus Rift, Available online: https://www.oculus.com/ (Accessed 10/06/2021)
30 Vuzix Star 1200 XL, Available online: https://www.slashgear.com/vuzix-star-1200-xl-see-
through-ar-headset-gets-even-more-immersive-18248074/ (Accessed 10/06/2021)
31 Vuzix Blade-Enterprize, Available online: https://www.vuzix.com/Products/Blade-Enterprise
32 Woodward, C., Kuula, T., Honkamaa, P., Hakkarainen, M., & Kemppi, P. (2014).
Implementation and evaluation of a mobile augmented reality system for building maintenance.
In Proceedings of the 14th International Conference on Construction Applications of Virtual
Reality, CONVR 2014 (pp. 306-315)
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 77
Division of Design & Manufacturing
33 Pintzos, G., Rentzos, L., Papakostas, N., & Chryssolouris, G. (2014). A novel approach for the
combined use of AR goggles and mobile devices as communication tools on the shopfloor.
Procedia CIRP, 25, 132-137. DOI: https://doi.org/10.1016/j.procir.2014.10.021
34 Mourtzis, D., Vlachou, E., & Zogopoulos, V. (2018). Mobile apps for providing Product-
Service Systems and retrieving feedback throughout their lifecycle: A robotics use case.
International Journal of Product Lifecycle Management, 11(2), 116-130. DOI:
https://dx.doi.org/10.1504/IJPLM.2018.092821
35 Mourtzis, D., Angelopoulos, J., & Panopoulos, N. (2020). A framework for automatic
generation of augmented reality maintenance & repair instructions based on convolutional
neural networks. Procedia CIRP, 93, 977-982. DOI:
https://doi.org/10.1016/j.procir.2020.04.130
36 8 Best augmented reality SDK for AR development for iOS and Android in 2017. (2017,
January). (ThinkMobiles) Retrieved September 2017, from https://thinkmobiles.com/blog/best-
ar-sdk-review/
37 Vuforia PTC. (2017). (Vuforia) Available online: https://www.ptc.com/en/products/vuforia
(Accessed 10/06/2021)
38 Wikitude SLAM Technology (2021), Available online: https://www.wikitude.com/wikitude-
slam/#:~:text=SLAM%20(simultaneous%20localization%20%26%20mapping),and%20Mark
er%2Dbased%20AR%20Projects (Accessed 10/06/2021)
39 Unity - Game Engine. (2021). Available online: https://unity3d.com/ (Accessed 10/06/2021)
40 Visual Studio IDE, Available online: https://visualstudio.microsoft.com/ (Accessed
10/06/2021)
41 Milgram, P., Takemura, H., Utsumi, A., & Kishino, F. (1995, December). Augmented reality:
A class of displays on the reality-virtuality continuum. In Telemanipulator and telepresence
technologies (Vol. 2351, pp. 282-292). International Society for Optics and Photonics.
42 Siltanen, S. (2012). Theory and applications of marker-based augmented reality: Licentiate
thesis.
43 Mourtzis, D., Angelopoulos, J., & Panopoulos, N. (2020). Intelligent Predictive Maintenance
and Remote Monitoring Framework for Industrial Equipment based on Mixed Reality. Frontiers
in Mechanical Engineering, 6, 99. DOI: https://doi.org/10.3389/fmech.2020.578379
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 78
Division of Design & Manufacturing
44 Goedkoop, M. (1999). Product service systems. Ecological and economic basis.
45 Tukker, A. (2004). Eight types of product–service system: eight ways to sustainability?
Experiences from SusProNet. Business strategy and the environment, 13(4), 246-260.
46 Shimomura, & Akasaka. (2013). Toward Product-Service System Engineering: New System
Engineering for PSS Utilization. In H. Meier, Product- Service Integration for Sustainable
Solutions (pp. 27-40). Springer.
47 Meier, H. V. (2011). Industrial Product-Service Systems (IPS2). The International Journal of
Advanced Manufacturing Technology, 52(9), 1175-1191.
48 Rese, M., Karger, M., & Strotmann, W. C. (2009). The dynamics of Industrial Product Service
Systems (IPS2) - using the Net Present Value Approach and Real Options Approach to improve
life cycle management. CIRP Journal of Manufacturing Science and Technology, 1(4), 279–
286. https://doi.org/10.1016/j.cirpj.2009.05.001
49 Mourtzis, D., Angelopoulos, J., & Panopoulos, N. (2020). Recycling and retrofitting for
industrial equipment based on augmented reality. Procedia CIRP, 90, 606-610. DOI:
https://doi.org/10.1016/j.procir.2020.02.134
50 EMCOR Group, Available online: https://www.emcorgroup.com/ (Accessed 10/06/2021)
51 SITEC, Available online: https://www.sitec-technology.de/home (Accessed 10/06/2021)
52 Pak, T. (n.d.). Tetra Pak ® Maintenance Services Proactive care to secure performance.
Available online: https://www.tetrapak.com/solutions/services/service-portfolio/maintenance-
services (Accessed 10/06/2021)
53 T. Mahboob, M. Z. (2016). Adopting Information Security Techniques for Cloud Computing -
A Survey. Proceedings - 2016 1st International Conference on Information Technology,
Information systems and Electrical Engineering, ICITISEE, (pp. 7-11).
54 Ayers, R. (2016, September 12). 5 Advantages of Using Encryption Technology for Data
Protection. (SmartDataCollective) Retrieved September 11, 2017, from
https://www.smartdatacollective.com/5-advantages-using-encryption-technology-data-
protection/
55 Rosato, D. V., & Rosato, M. G. (2012). Injection molding handbook. Springer Science &
Business Media.
DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM
BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis
Department of Mechanical Engineering & Aeronautics 79
Division of Design & Manufacturing
56 Meier, H., Lagemann, H., Morlock, F., & Rathmann, C. (2013). Key performance indicators
for assessing the planning and delivery of industrial services. Procedia CIRP, 11, 99-104. DOI:
https://doi.org/10.1016/j.procir.2013.07.056
57 Mourtzis, D., Boli, N., & Fotia, S. (2017). Knowledge-based estimation of maintenance time
for complex engineered-to-order products based on KPIs Monitoring: a PSS Approach.
Procedia CIRP, 63, 236-241. DOI: https://doi.org/10.1016/j.procir.2017.03.317
58 Mourtzis, D., Fotia, S., & Vlachou, E. (2017). Lean rules extraction methodology for lean PSS
design via key performance indicators monitoring. Journal of manufacturing systems, 42, 233-
243. DOI: https://doi.org/10.1016/j.jmsy.2016.12.014
59 Karrach, L., Pivarčiová, E., & Božek, P. (2020). Identification of QR Code Perspective
Distortion Based on Edge Directions and Edge Projections Analysis. Journal of Imaging, 6(7),
67. DOI: https://doi.org/10.3390/jimaging6070067
60 Wada, T., Kishimoto, Y., Nakazawa, E., Imai, T., Suzuki, T., Arai, K., & Kobayashi, T. (2020,
October). Multimodal User Interface for QR Code based Indoor Navigation System. In 2020
IEEE 9th Global Conference on Consumer Electronics (GCCE) (pp. 343-344). IEEE. DOI:
https://doi.org/10.1109/GCCE50665.2020.9292026
61 The Base16, Base32, and Base64 Data Encodings. (2006). (The Internet society) Available
online: https://www.rfc-editor.org/rfc/pdfrfc/rfc4648.txt.pdf (Accessed 10/06/2021)