A Method for Evaluating End-User Development Technologies

Americas Conference on Information Systems,

AMCIS 2017, Aug, 10th

A METHOD FOR EVALUATING END-USER DEVELOPMENT TECHNOLOGIES

Prof. Dr. Claudia Melo, Dep. of Computer Science Jonathan Moraes, Fac. of Software Engineering Marcelo Ferreira, Fac. of Software Engineering

Prof. Dr. Rejane Figueiredo, Fac. of Software Engineering University of Brasília, Brazil

WHY ARE WE INVESTIGATING EUD TECHNOLOGY EVALUATION?

• # of end users > # of professional so1ware developers • > 50M end-user developers,

USA only (Burne9 and Myers, 2014)

• Li9le research has empirically examined so1ware package evalua7on criteria and techniques in general, and in the end-user development context in par7cular (Harnisch, 2014; Jadhav and Sonar, 2009; Jadhav and Sonar, 2011; Misra and Mohanty, 2003)

• Success and failure of end-user development within an organizaOon ulOmately depends on how effec7ve so1ware packages are chosen and used

2

• This research study is part of a partnership with the Ministry of CommunicaOons, Science, and Technology, Brazil

• Plenty of EUD tools being offered in the market

• OrganizaOons sOll lost on how to take full advantage of EUD (facing cultural, managerial and technological challenges). But already buying soluOons!

INDUSTRY CONTEXT

3

Three different areas have important, but parOal, contribuOons to our research purpose:

•So#ware package acquisi1on research;

•So#ware quality models & CSCW/HCI research, and

•Technology acceptance research.

LITERATURE REVIEW

4

WHAT WAS CONSIDERED AS A STARTING POINT?

The model should evaluate technologies that have: •Essen1al quali1es that enable the end-user developer to manipulate the

tool and produce useful results in a certain applica1on domain •from so]ware quality models and CSCW/HCI models;

•General quali1es inherent to so#ware packages •from so]ware package acquisiOon models and technology acceptance models;

•Essen1al quali1es for management and technological governance •from so]ware package acquisiOon models;

•An evalua1on method based on already-established and tested techniques, even if they come from a different context

5

• Evalua1on criteria, characteris1cs, sub characteris1cs, and aJributes

• Developed ques1ons and metrics from different points-of-view (PoV).

STRUCTURING THE EVALUATION MODEL

6

THE EVALUATION METHOD

1. Functional characteristics:

○Functionality (3 questions, PoV: Governance/Platform)

○Collaboration (13 questions, PoV: EUD/Platform)

○Data Management (4 questions, PoV: EUD/Platform & 4 questions, PoV: Governance/Platform)

2. Cost and Benefit Characteristics (6 questions, PoV: Governance/Platform)

3.Vendor Characteristics (4 questions, PoV: Governance/Platform)

4. Software Quality Characteristics

○ Compatibility (6 questions, PoV: EUD / Platform)

○ Maintainability (6 questions, PoV: EUD / Platform)

○ Usability (212 questions, PoV: EUD / Platform)

○ Reliability (7 questions, PoV: IT Manager/Platform)

○ Performance Efficiency (6 questions, PoV: EUD / Platform)

○ Security (5 questions, PoV: Governance/Platform & 1 question, PoV: EUD/Platform/Developed Application & 9 questions, PoV: EUD / Platform)

5.Hardware & Software Configuration (8 questions)

The complete EUD technology evaluation model is available at: https://itrac.github.io/eud_technology_evaluation7

https://itrac.github.io/eud_technology_evaluation

EVALUATING EUD TECHNOLOGIES - STEPS

1. Determining the need, including high-level investigation of software features and capabilities provided by vendors August/2016 to October/2016 - literature review and contact with leaders of public and private organizations to build a general list of tools.

2. Short listing candidate packages and eliminating the candidate packages that do not have the required feature We shortlisted the most solid market offers (Forrester, 2016).

8

3. Using the proposed evaluation technique to evaluate remaining packages and obtain a score

4. Pilot testing the tool in an appropriate environment (parallel to 3) Fundamental to refine the model proposed (removing, rewriting, and adding questions/metrics).

The evaluation model and the platform evaluation results presented in this work are already the result from a second evaluation iteration

EVALUATING EUD TECHNOLOGIES - RESEARCH STEPS

9

EXECUTION DETAILS

The platforms:

● OutSystems

● Oracle Apex

● Salesforce Lightning¹

● Zoho Creator¹

1 - The Salesforce Lightning and Zoho Creator evaluations are not described in the paper.

10

Testing:

• 3 software engineering students as testers;

• Each student performed the entire model for every platform using four default scenarios (C-R-U-D)

• Either using the platform’s predefined templates or not.

• A researcher supervised the testers and validated the forms.

EXECUTION DETAILS

11

Results: Functional Characteristics

Functionality

Sub-Characteristics Oracle Apex OutSystems Salesforce Lightning Zoho Creator

Application Domain Database General-purpose General-purpose Database

Collaboration

Shareability ✔ ✔ ✔ ✔

Coordination of Actions ✔ ✔ ✔ ✘

Consequential Communication

60% 60% 60% 80%

12

Results: Functional CharacteristicsCollaboration


Finding Collaborators and Establishing Contact

75% 0% 50% 100%

Concurrent Protection 0% 50% 0% 50%

Data Management

Data Input and Output 100% 100% 100% 66%

Required Technical Knowledge

33% 33% 33% 0%

13

Results: Cost and Benefit & Vendor Characteristic

Cost


License Cost $ 164,839.00 $ 2,072,601.74 $ 3,600.00 per user/year

$ 1999.00 per month

Maintenance Cost N/A N/A N/A N/A

Vendor

Contract Dependency 100% 100% 100% 100%

Technology Dependency

100% 0% 0% 100%

14

Results: Software Quality Characteristics

Compatibility


Technical Knowledge Requirement

Advanced Advanced N/A Average

Data Exchangeability ✘ ✔ ✘ ✘

Connectivity With External Component/System

Possible Possible Impossible Possible

Reusability ✔ ✔ N/A ✔

15


Maintainability


Modifiability 100% 100% 50% 100%

Reusability Possible, Easy Possible, Easy Possible, Average Possible, Hard

Reliability

Availability N/A N/A N/A N/A

Vendor Support 0.41 fixes/day 0.43 fixes/day 0.12 fixes/day 0.73 fixes/day

16


Performance Efficiency


Response Time 100% 100% 100% 100%

Turnaround Time 50% 100% 75% 100%

Security

Access Behaviors 80% 60% 60% 20%

Developed Application Security

100% 100% 100% 0%

17


Security


Update Behaviors 100% 50% 100% N/A

File Upload Security 100% 100% 100% 100%

Report Behaviors ✔ ✔ ✘ ✘

Security Algorithms 50% 100% 100% 50%

18

Results: Usability

19

CONCLUSION

● The major original contributions of the paper are:

○ A detailed method for evaluating EUD technologies that comprises 11 characteristics, 20 sub-characteristics, 30 attributes, 300 questions/metrics, and

○ Evaluations using the method against 4 leading EUD platforms in the market (we described only 2 in the paper)

○ Next steps: validation with real-world scenarios (e.g. action research) & investigation of automation opportunities

20

Questions? Thanks!

[email protected]

Claudia Melo21

Acknowledgement: this research received support from

Fundação de Apoio a Pesquisa do Distrito Federal (FAP-DF)