Survey on the text entry methods used in touch screen ...scopus)1.pdfentry methods used in touch screen mobile devices by visually challenged’ presented at International Conference

Int. J. Advanced Intelligence Paradigms, Vol. X, No. Y, xxxx 1

Copyright © 20XX Inderscience Enterprises Ltd.

Survey on the text entry methods used in touch screen mobile devices by visually challenged

M. Shabnam* and S. Govindarajan Department of Computer Applications, SRM University, Kattankulathur, India Email: [email protected] Email: [email protected] Email: [email protected] Email: [email protected] *Corresponding author

Abstract: With the onset of electric and electronic technology, the world has taken a metamorphic change from the mammoth size of an elephant to that of a tiny ant. The coaxial cable bridges encircle the spear which generates them. By far, the mobile phones have been the most advanced invention of the human race, and they have contributed an incredible part in the lifestyle of people of all walks of life. The handy phones (Clarridge, 2009) have succeeded in communication, kindred activities. There is a host of methods by which the visually challenged people use the mobile for their multifarious activities. Our focus here is to recount salient features of diverse interaction approaches followed by this vulnerable segment of society to interact with touch-based mobile phones. Thus, in this document, an earnest effort is made to assess various interaction, methods in terms of several features like the implication of interaction design, design philosophy, concrete usability study measures, favourable factors, divergence in utilisation so as to put them in a nutshell. A case study on the interaction approaches of the visually challenged with mobile phones is also offered here along with an evaluation of the text input time for several text input techniques, though as per the observation there is no major divergence in the text input time.

Keywords: survey on interaction techniques; Braille coded text entry; comparative study; touch screen text entry; text entry in mobile; text entry by blind; visually impaired.

Reference to this paper should be made as follows: Shabnam, M. and Govindarajan, S. (xxxx) ‘Survey on the text entry methods used in touch screen mobile devices by visually challenged’, Int. J. Advanced Intelligence Paradigms, Vol. X, No. Y, pp.xxx–xxx.

Biographical notes: M. Shabnam obtained her Bachelor’s degree in Computer Science from Bharadhidasan University in 2001. Then she obtained her Master’s degree in Computer Science and is pursuing PhD in Computer Science majoring in human computing interaction, and gesture computing from SRM University, India. Currently, she is an Assistant Professor at the Faculty of Science and Humanities, SRM University, India. Her specialisations include Gesture computing, HCI, cloud computing technologies. Her current research interests are analysing the blind users’ interaction with various types of mobile designs, research on Braille code and developing a gesture-based interaction methodology for visually impaired users, big data, cloud security management.

2 M. Shabnam and S. Govindarajan

S. Govindarajan received his BSc degree from Madras University, MSc in Engineering from PSG college of Technology, Coimbatore, and PhD in Information Technology (IT) from Faculty of Engineering Madras University. Earlier, he had served major software MNC as General Manager. Currently, he works for SRM University as a Professor and Head for Computer Applications Department (PG) since 2009. He has published more than ten journals, which include five international publications. His research interests are HCI, ERP and software processes.

This paper is a revised and expanded version of a paper entitled ‘Survey on text entry methods used in touch screen mobile devices by visually challenged’ presented at International Conference on Intelligent Computing and Data Science 2014, Thanjavur, Tamil Nadu, 12–13 September 2014.

1 Introduction

In view of the mounting nature of the mobile devices in terms of computational supremacy and capacity, they are competent to displace or renovate the modern communication methods in areas where the communication technology performs in an inferior manner. The Mobile platforms bring in more novel design issues (Evreinova, 2005) than classically faced while devising their desktop counterparts (Hooten et al., 2013). The amazing quality of a mobile device is that when it is linked to a wireless network, it is very easy for us to access data from any nook and corner of the cosmos 24/7. Nevertheless, plagued by the tiny screens and diverse input techniques, the mobile devices at times turn to be not user-friendly, notably for accessing the net (Roudaki et al., 2014). Further, as they are habitually on movement, it becomes very tough to limit physical access, placing greater dependence on the system employed in the device itself. Deployment of the novel device in manifold scenarios tends to enhance the fears regarding the safety concerns to the effect that the password, PIN or sketch may be subjected to unauthorised scrutiny during entry, endangering the aspect of secrecy (Clarridge, 2009).

The modern investigations have mainly focused on comprehending the intention contained in the text or alternately the text identified from a piece of voice. Another option is to leverage speech detection methods to prop up voice as an input (Zhang et al., 2015). The touch screen technology has won wide acclaim from the people on account of its flexibility in design, swiftness and expediency. In regard to touch screen use, it is observed that the clients experienced higher levels of subjective distress with touch screen use when compared with keyboard use (Amend-Irwin and Sesto, 2012). However, the touch screen-based smart phones are well-geared to successfully tackle these menaces.

First, the finger-based instinctive communications are found to be very fruitful, whatever be the age of the user. Second, touch screen-based interfaces permit us to have senior-friendly interfaces by means of software-level adaptations (Kobayashi et al., 2011). Touch-based mobile devices are equipped with common traits such as complete browsing or smart functions. Consequently, a significant investigation was conducted which concentrated on input methods to augment the precision of the touch usability analyses for touch screen keyboards and evaluations of the input offset (Jung and Im,

Survey on the text entry methods used in touch screen mobile devices 3

2015). The text conclusion time and the number of inaccuracies were evaluated, and words per minute and keystrokes per minute were computed to assess the performance among the text entry tasks (Ryu et al., 2013). As the input required numerous or recurrent presses, the key strokes per character (KSPC) were enhanced. In addition to the decrease in its input efficiency reduced, it resulted in repetitive strain injuries (RSI) to the hand (Wu et al., 2014). However, the present-day mechanisms fail to support mobile text entry effectively in view of the limitations in dimensions of the mobile devices. Nowadays, almost all the phones are fitted with very many buttons, either physical or virtual and are very minute in size. And the large majority of users are forced to glance through their device while entering texts in to it (Southern et al., 2012).

The interaction techniques are analysed based on the factors such as

a implications for design

b design principles

c actual usability study

d beneficial factor

e conflict in usage.

1.1 Implications for design

Implications for design are the common factor for different interaction techniques compared in this paper; hence each section does not include ‘implications of design’ part separately. The text input techniques and mobile interaction approaches were devised for the sake of convenience for the visually challenged brethren so that they can effortlessly and effectively exploit the unique characteristics of the mobile phone. There has been a phenomenal increase in the number of touch-based mobile phones over the last decades. Thus it becomes all the more essential to launch novel text entry approaches in the touch version of the mobile phone (Lagoa et al., 2007).

In this connection, the slide rule defines the fundamental gestures such as the tapping, sliding from left to right, double tapping to access the mobile apps in which a satisfactory solution is achieved to address the issue of button-based apps access. The adaptive blind interaction approaches for the touch screens were spearheaded to cutback the screen spatial hassle in the mega screen layout in the mobile phone screens.

Further, the no-look notes furnish a divergent category of screen layout which has come out with flying colours in overwhelming the malaises such as the inaccurate identification of the letters concerned. For example, m and n have identical sounds and so do c and z which is likely to confound the user prompting him to make fallacies in the appropriate choice of the exact text in the screen. Thanks to the no-look notes layout design, the relevant paradox has been satisfactorily addressed.

BloNo (Kane and Wobbrock, 2011): a novel mobile text entry interface for the visually challenged has envisioned an interface design which is competent to usher in quickest text entry time-frame. However, it is essential to employ two fingers in this interface design so as to input a character.


2 Slide rule interaction technique-using multi touch screen

2.1 Design principles

The slide rule interaction technique was launched to access the mobile touch screen objects by the blind people. It defines gestural actions to carry out the functions like placing a phone call, reading an e-mail message, playing a song with audio feedback, even though explicit gestures intended to create a text entry are conspicuous by their absence (Kane et al., 2008). It is true that the slide rule (Lenman et al., 2002) effectively facilitates object choice in the touch mobile device, but it fails to include any technique for the purpose of making the text entry. In fact, the Screen is empty, having only the sound feedback, and the objects are listed on by one, and the choice of object is carried out by means of the right-flick gesture devoid of the tap, which has the effect of cutting down the time involved in continuously pointing a position on the screen to choose an object. The dimension of the objects on the screen is set 50.8 mm by 7.62 mm. Further, to make a call the user has to flick right on the touch screen, the system replies the mail object, to choose the corresponding mail the user has to tap the screen. In an identical way, he had to flick down to distinguish the menu items, followed by flicking right to choose the relative menu item. This is how the gestures are deployed to access the mobile objects.

Figure 1 Slide rule flicks (see online version for colours)

(a) (b)

(c) (d)


2.2 Actual usability study

The slide rule interaction technique was experimented with Apple iPhone and then analysed and contrasted with the ASUS MyPal pocket PC A730. It was found that the contributors were susceptible to committing a multitude of mistakes while employing the iPhone. In this regard, the dependent variables taken into account included the task completion duration, task completion flaws, the listening period per item in the investigation to analyse the efficiency in accomplishment of the button-based and slide rule interaction techniques. The charismatic test outcomes illustrated the fact that the task conclusion duration of the slide rule interaction mechanism was significantly lesser than the button-based interaction system, though the latter was preferred by the users, thanks to its familiarity.

2.3 Beneficial factor

The paper slide rule was able to usher in various glittering advantages as detailed below.

1 the ‘gesture operation at finger resolution not screen resolution’ – was more user-friendly and therefore was widely accepted

2 the ‘reduced demand for selection accuracy’ – went a long way in facilitating user flexibility

3 the ‘quick browsing and navigation’ – was able to incredibly cut down the access time, thus delighting the user

4 the ‘intuitive gestural mappings’ – contributed a lot toward the user flexibility.

Further, the user response made it crystal clear that though he voted for the button-based interaction for its vast familiarity and his preference for the slide rule stemmed from its easy accessibility on touch screen.


The paper slide rule was able to usher in various glittering advantages as detailed below.

1 the ‘gesture operation at finger resolution not screen resolution’ – was more user-friendly and therefore was widely accepted

2 the ‘reduced demand for selection accuracy’ – went a long way in facilitating user flexibility

3 the ‘quick browsing and navigation’ – was able to incredibly cut down the access time, thus delighting the user

4 the ‘intuitive gestural mappings’ – contributed a lot toward the user flexibility.

Further, the user response made it crystal clear that though he voted for the button-based interaction for its vast familiarity and his preference for the slide rule stemmed from its easy accessibility on touch screen.


2.5 Conflict in usage

The slide rule is found to be very convenient for the touch screen scenario, though there crops a mounting number of mistakes on account of the fact it is a new device, and therefore, the users are prone to committing errors. The continuous application of the slide rule technique is expected to scale down the error rate involved in the selection of an object.

3 Adaptive blind interaction techniques for touch screens


As per the layout of the touch screen, the pie menu of alphabets is organised as layers; the user has to always place his finger on the screen, so as to navigate through the pie menu of characters. The character is chosen at the point where the user takes away his finger from the pie menu. This interface design is designed in accordance with the nav-touch.

Figure 2 Adaptive blind interaction interfaces


The test outcomes threw open the potential use of the system in parallel touch-based PDAs. The design is intended solely for the visually challenged people, and is consistent and vigorous. The mixture of gestural patterns and hardware buttons were devised for the choice of text in the screen, which needs only a miniature screen layout design, thereby eliminating the screen spatial abnormalities.


The adaptive design method enabled several users carry out the text entry at a speed of 12 wpm.


3.4 Conflict

Dwell time is incorporated in the text entry time which represents the duration of time needed to intersperse the layers of the three sets. For the purpose of entering just character, the user has to utilise both hands, which is not practical when they are busy with other duties.

4 No-look notes


No-look notes elegantly organise characters around the screen in an eight-segment pie menu evocative of the mechanisms launched by Y Fantidis and Raman. Nevertheless, each section of the menu is home a multitude of characters, in such a way that all the letters of the English alphabet make their august appearance. The segments represent soft-buttons which need to be pressed for activation.

Figure 3 No-look notes layout (see online version for colours)


The document evaluated the features of the no-look notes and the voice-over, and there was considerable divergence between both of them, as per the mathematical report, with just one partaker achieving an excellent score for the no-look notes and the average text entry speed was found to be 1.67 wpm.


The prominent distinction in the utilisation of Apple Voice over and no-look notes included various features such as the ease of operation, the swiftness of execution, amazing quality of control, simplicity of comprehension to learn which were ostensible and which when augmented with practice, facilitated the function of entering text accessible.


5 BloNo: a new mobile text-entry interface for the visually impaired


The alphabet was segregated into five components, each starting with a dissimilar vowel, as it was easy to remember them quite distinct from other randomly selected letters. The corresponding components were subsequently utilised to generate the o create character matrix Here, the keyboard keys are deployed as the joystick to navigate through the matrix. Exploiting all the directions empowers the user to cut down the number of key presses required to arrive at a particular letter.

Figure 4 Mobile text entry interface access way (see online version for colours)

5.2 Beneficial factors

The supplementary traits like the automatic word finishing (AWF); task aware dictionaries significantly improve the function of text entry with the novel interface design. From among the five visually challenged participants, one was competent to send a SMS by means of the novel interaction system.

5.3 Conflict

The vowel driven technique necessitates memory skills to memorise the character organisation in the novel interface design. The speed between the three layers of alphabetical pie arrangement is found to be hard for the blind folders.

6 Braille touch (Frey et al., 2011): mobile touch screen text entry for the visually impaired


The standard Braille keyboard interface was implemented on the touch screen (MacKenzie and Soukoreff, 2002); and the user is allowed to use three fingers such as index finger, middle finger, and ring finger of both the hands to touch the Braille touch interface where the operation are space, backspace, enter can be done by the gestures right-flick, leftflick, two finger right-flick.


6.2 Study design

The researchers were experimented typing performance of blind users in three ways, they are

1 Braille touch on experimental Braille keyboards

2 a soft Braille keyboard on a touch screen tablet

3 and PACmate, a commercial Braille keyboard with hard buttons.

6.3 Study results

The blind users are segregated into three different groups. The first group consists of experts in Braille performance, second group consists of moderate Braille performers, and the third group consists of poor Braille knowledge people. Table 1 states the (McGookin et al., 2008) Mackenzie and Soukoreff’s total error rate. The results observation proves there is no much more difference in between two touch screens, but significant change is obtained in between touch screens and PAC mate. The authors are researching further error types being committed by the visually challenged people (Yfantidis and Evreinov, 2005). Table 1 Braille touch: mobile touch screen text entry for the visually impaired

Error rate Text entry rate

Tablet Smart phone PACmate Tablet Smart phone

Group 1 14.8% 14.5% 5.3% 24.2wpm 23.2wpm Group 2 40.5% 33.1% 11.0% 18.2wpm 21.0wpm Group 3 45.0% 39.3% 19.3% 8.3wpm 9.4wpm

6.4 Beneficial factors

Compared with the touch screens visually impaired are performing well with physical button phones. The number of error in PACmate is comparatively less. The following table illustrates the mobile interaction explicit traits and test participant data. Table 2 Text entering time by different interaction methods based on literatures

Methods name No. of users in tests User’s age group Text entry measure

Slide rule interaction technique

10 31.4 11.69 secs (object selection time)

Adaptive blind interaction technique

12 24–35 12 WPM

No-look notes 10 40.8 1.67 WPM A new mobile text-entry interface

8 21–64 2 WPM

Braille: mobile touch screen text entry for the visually impaired

11 -- 18–24 WPM


Figure 5 Text input time by different interaction method (the higher is better) (see online version for colours)

7 Case study on text entry on button-based, and touch screen

There was a concurrent case study for the mobile accessibility by visually challenged people carried out in and around Chennai. Almost 60 persons were interviewed at various locations at Chennai. The visually challenged people were presented a questionnaire regarding the following features.

• name, age group, hometown

• literacy level, braille literacy

• applications habitually employed in mobile,

• type of mobile utilised

• interaction method employed

• text input technique followed.

Figure 6 Soft keyboard-touch screens (see online version for colours)

The case study revealed the fact that about 40% of visually challenged people employ touch screen mobile phones and the rest utilise the button-based mobile device. 20% of people use the Apple iPhone, 18% employ Samsung galaxy, 2% deploys parallel touch mobile phones and 50% people utilise the Nokia button-based mobile phones.


7.1 On the spot test

The survey was conducted by means of an instant test with an individual or a group of individuals interviewed at the same instant. The underlying intention of the test was to locate the text input time with their customised interface design. The participants were categorised according to the age group and the interaction methods employed by them. The age groups are effectively exhibited in Figure 7.

Figure 7 Text input time by participants (see online version for colours)

The interaction techniques can be generally categorised into two classes like the touch-and-tell approach and the button-based technique. It is crystal clear from the figure that the text input period changes in between the users and the techniques.

7.2 Result analysis

The observations are recorded and scientifically evaluated. The two-pronged ANOVA (RR.Wilcox, 2002) technique illustrates the null hypothesis ‘H0: the thinking that there exists a considerable distinction between the visually challenged individuals in employing the mobile interaction techniques’ is erroneous. The estimated value Fo falls within the critical value Fe in between the age groups. In between the interaction techniques the estimated value Fo is within the critical value Fe. This phenomenon implies that there exists no considerable distinction e in between persons in regard to the usage of several interaction techniques, and this holds good for diverse age groups of visually challenged people. Table 3 Case study – statistical report

Variance DF SS MS F0 Fe

Between age groups 2 31 15.5 0.67 9.55 Between interaction methods 1 10.7 10.7 0.46 18.51 Error 2 46.3 23.15

8 Conclusions and future enhancement

The renovation of Mobile interaction techniques (RR.Wilcox, 2002) for the utmost mobile usage of visually challenged people has become one of the most-modern upcoming research topics. With the intention of augmenting the slide rule technology for accessing the touch screen objects, the mobile interaction literatures have been subjected


to appraisal and the text input period evaluated by diverse techniques. The slide rule technique encompasses three fundamental gestures such as flick right, flick left, flick up, flick down, tap, double tap to access the applications on the screen. The slide rule is mainly concerned with the mobile object accessibility, but has nothing to do with the text entry technique. The purpose of our investigation to define a novel structure of gestural models so as to write text in touch-based mobile device. The slide rule basic flick gestures are well-known to the blind people as revealed by our case study reports, Therefore, we intend to deploy the fundamental gestures in order to write characters in the touch screens without just navigating a specific character from the screen, which ultimately results in significant consumption of time for the object selection. In addition, the concurrent case study also enables us to assess the real statistics of mobile usability by the visually challenged people.

References Amend-Irwin, C. and Sesto, M. (2012) ‘Performance and touch characteristics of disabled and

nondisabled participants during a reciprocal tapping task using touch screen technology’, Journal of Applied Ergonomics, Vol. 43, No. 6, pp.1038–1043.

Clarridge, C. (2009) ‘New device lets deaf-blind people converse with anyone’, Seattle Times. Czajkowski, K., del Rey, M., Fitzgerald, S., Foster, I. and Kesselman, C. (2001) ‘Grid information

services for distributed resource sharing’, Proceedings of the 10th IEEE International Symposium on High Performance Distributed Computing, INSPEC Accession No. 7062660, IEEE Transactions on Rehabilitation Engineering, pp.81–94.

Evreinova, Y.G. (2005) ‘Alternative visualization of textual information for people with sensory impairment’, Academic Dissertation, pp.1–85.

Frey, B., Southern, V. and Romero, M. (2011) ‘BrailleTouch: mobile texting for visually impaired’, Universal Access in Human Computer Interaction with Mobile Devices and Services, Vol. 6767, pp.19–25.

Hooten, E., Hayes, S. and Adams, J. (2013) ‘Communicative modalities for mobile device interaction’, Journal of Human-Computer Studies, Vol. 71, No. 10, pp.988–1002.

Jung, E. and Im, Y. (2015) ‘Touchable area: an empirical study on design approach considering perception size and touch input behavior’, Journal of Industrial Ergonomics, Vol. 49, pp.21–30.

Kane, S. and Wobbrock, J. (2011) ‘Usable gestures for blind people: Understanding preference and performance’, Proceedings of SIGCHI Conference on Human Factors in Computing in Computing Systems, pp.413–422.

Kane, S., Bigham, J. and Wobbrock, J. (2008) ‘Slide rule: making mobile touch screens accessible to blind people using multi-touch interaction techniques’, Assets 08, pp.73–80.

Kobayashi, M., Hiyama, S., Miura, T., Asakawa, C., Hirose, M. and Ifukube, T. (2011) ‘Elderly user evaluation of mobile touch screen interactions’, in Campos et al. (Eds.): INTERACT 2011, Part I, LNCS 6946, pp.83–99, ©IFIP International, Federation for Information Processing.

Lagoa, P., Santana, P., Guerreiro, T., Gonçalves, D. and Jorge, J. (2007) ‘BloNo: a new mobile text-entry interface for the visually impaired’, Universal Access in HCI, Part II, LNCS 4555, pp.908–917.


Lenman, S., Bretzner, L. and Thuresson, B. (2002) Computer Vision Based Hand Gesture Interfaces for Human-Computer Interaction, pp.1–9, Report No.: TRITA-NA-D0209, CID-172, Department of Numerical Analysis and Computer Science, ISSN 1403-0721 (print) 1403-073.

MacKenzie, S. and Soukoreff, W. (2002) ‘Text entry for mobile computing: models and methods, theory and practice’, Human-Computer Interaction, Vol. 17, No. 2, pp.147–198.

McGookin, D., Brewster, S. and Jiang, W.W. (2008) ‘Investigating touchscreen accessibility for people with visual impairments’, Conference on Human-Computer Interaction, pp.298–307.

Roudaki, Kong and Yu, (2014) ‘A classification of web browsing on mobile devices’, Journal of Visual Languages and Computing, February, Vol. 26, pp.1–28.

Ryu, T., Lim, J., Song, J., Yun, M.H. and Shin, M. (2013) ‘Performance comparison between the preferred right and preferred left hands in text entry using qwerty touch keyboard smart phones’, Journal of Industrial Ergonomics, Vol. 43, No. 3, pp.400–405.

Southern, C., Clawson, J., Frey, B., Abowd, G. and Romero, M. (2012) ‘An evaluation of Braille touch: mobile touch screen text entry for the visually impaired’, MobileHCI ‘12 Proceedings of the 14th International Conference on Human-Computer Interaction with Mobile Devices and Services, ACM, New York, mpp.317–326.

Wilcox, R. (2002) ‘Understanding the practical advantages of modern ANOVA methods’, Journal of Clinical Child and Adolescent Psychology, Vol. 31, No. 3, pp.399–412.

Wu, F-G., Huang, Y-C. and Wu, M.L. (2014) ‘New chording text entry methods combining physical and virtual buttons on a mobile phone’, Journal of Applied Ergonomics, July, Vol. 45, No. 4, pp.825–832.

Yfantidis, G. and Evreinov, G. (2005) ‘Adaptive blind interaction technique for touchscreens’, Universal Access in the Information Society, University of Computer Science, May 2006, Vol. 4, No. 4, pp.328–337.

Zhang, N., Mei, T., Hua, X-S., Guan, L. and Li, S. (2015) ‘TapTell: interactive visual search for mobile task recommendation’, Journal of Visual Communication Image Research, May, Vol. 29, pp.114–124.

Documents

Survey on the text entry methods used in touch screen ...scopus)1.pdfentry methods used in touch screen mobile devices by visually challenged’ presented at International Conference