1  

PAPELUCHO: A MODEL FOR DEVELOPING INCLUSIVE DIGITAL BOOKS FOR DEAF CHILDREN Authors: Ricardo Rosas, Soledad Véliz, Rodrigo Arroyo, María Ignacia Sánchez, Marcelo Pizarro, Andrés David Aparicio Affiliation: Center for the Development of Inclusion Technologies (CEDETi UC), Pontificia Universidad Católica de Chile; Santiago, Chile. Abstract In general, deaf and hard of hearing people (D/HH) must overcome several obstacles to acquire the reading-writing code. In this context, CEDETi UC created a model for developing inclusive digital books for deaf children which was first implemented through an adaptation of the title “Papelucho”. The purpose of this media is to support the reading comprehension of D/HH children. This study explores the reading comprehension of deaf students aged 9 to 14 who experience a book in three modes: text only, Sign Language only, and multimedia (text, images, and Sign Language). The research analyses how each type of content and their interaction enhance reading comprehension in Chilean D/HH children. Variables that affect reading skills, such as vocabulary, were also considered. The hypothesis advanced is that this type of multimedia mode allows deaf children to approach literacy more effectively. Introduction For deaf people, the acquisition of the reading-writing code is fraught with difficulties. In this context, CEDETi UC created a model for developing inclusive digital books for deaf children which started with the implementation of an adapted version of “Papelucho”, a traditional Chilean book about the adventures of an 8-year-old child. This book, which includes the adapted text, Sign Language, images, and audio, allows D/HH children to explore it in a multimodal way. Users can access the digital book for free (http://papelucho.cedeti.cl/).

Figure 1. Papelucho’s accessible multimedia book Although reading and writing is an essential ability for individual development, many deaf or hard of hearing people (D/HH) have not achieved reading comprehension proficiency by the time they graduate from high school (J. L. Luckner & Handley, 2008). These difficulties of D/HH students can be explained, in part, by limited access to written language learning opportunities and to the fact that acquiring the written code involves learning a different language from the one they learn naturally: Sign Language (Meier, 1990; Supalla, 1991). Some studies on D/HH people attribute reading comprehension performance to many variables and difficulties such as reading skills, vocabulary, or problems with syntax processing (Kelly & Gaustad, 2007; Paul, 1998). Limited research has been conducted, both in general and with D/HH subjects, to determine the impact in comprehension when the text

2  

is not exclusively constructed by written words (Nikolaraizi, Vekiri, & Easterbrooks, 2013). We know that Sign Language, when used along with reading, increases the literacy levels of deaf children (Wilbur, 2000). D/HH students have been considered strong visual learners (Dowaliby, 1999). Observations in educational settings reveal greater visual sensitivity in this group (Marschark & Hauser, 2012). Therefore, D/HH people, whether they use oral or Sign Language, and whether they have better or similar visual skills compared with hearing people, must be ensured unrestricted access to visual or non-verbal information (Nikolaraizi et al., 2013) in the form of either Sign Language or images, which can be used as an asset to support their reading process. In this context, the present study highlights that a carefully designed multimodal device, which conveys information through images, text, and videos that complement each other, can support reading comprehension in D/HH students. It has been argued that multimodality can affect learning, which occurs when students receive information presented in more than one mode: verbal and non-verbal (Mayer, 1997). Research in multimodality applied to narrative texts suggests that, when knowledge is represented and communicated by multiple semiotic ways, meaning is distributed through those modalities. Therefore, different aspects of the text's meaning are carried by each modality, complementing and affecting one another (Daly & Unsworth, 2011). Multimedia tools have provided some evidence that, when text, pictures, videos in Sign Language, graphics, and animation are effectively combined, a positive impact is generated, which supports the comprehension of D/HH students (Mich, Pianta, & Mana, 2013). Previous research has assessed the impact on comprehension of different stimuli combinations such as Sign Language+Images, Sign Language+Text, and Sign Language+Text+Images (Gentry, Chinn, & Moulton, 2004). In this document, an accessible multimedia book will be understood as a digital resource that delivers information using verbal (text and Sign Language) and non-verbal (static images) modes, and which involves different presentation modes (images, videos, and text). A connection has been established between reading comprehension and vocabulary knowledge in D/HH children (Loeterman, Paul, & Donahue, 2002). Deaf children are reportedly delayed in their acquisition of vocabulary knowledge when compared to their hearing peers, which affects their comprehension levels (Allen, 1994; J. Luckner & Cooke, 2010). This research explores the relation between variables related to the text, in this case a multimodal device, and those associated with the reader, controlling for relevant individual variables such as vocabulary. Method The creation of this accessible multimedia book model consisted in the development of an innovative tool that allows technological mediation to stimulate literacy and comprehension in an inclusive way. In the book, users find images, adapted text, Sign Language videos, and audio. In order to understand the impact of reading in different modalities (text only, Sign Language, or multimodality, which includes text+Sign Language+images), a study was carried out at a Chilean Sign Language (ChSL)/Spanish bilingual school for D/HH children located in Santiago, Chile. It was an analytic study with repeated measures that sought to establish the relationship between modality of presentation and reading comprehension in D/HH children.

3  

28 children aged 9 to 14 students at a special education school were invited to the study. The participants had no additional documented special educational needs and their caretakers were asked to sign an informed consent. Six participants were removed from the sample due to application complications and non-compliance with the inclusion criteria. The final sample consisted of 22 children. This research involves a digital platform hosting the same content in three stimuli combinations: text only (TXT), in which the content was written in Spanish; Chilean Sign Language only (SL), in which the content was expressed through videos with a Chilean Sign Language interpreter; and the full version (FULL), in which the content was expressed both through text with Chilean Sign Language (ChSL) videos and through images with ChSL videos. The FULL version is different from the TXT and SL versions because it allows the reader to choose between two illustrated versions, with text or ChSL, of the same content. All three versions (TXT, SL and FULL) are based on the same text. The participants were randomly assigned to 6 groups, each of which was exposed to three versions (TXT, SL, and FULL) of the same story. The same comprehension questions were asked to participants after each exposure to the content. For example, children in the first group would read the story in the following order: 1) TXT, 2) questions, 3) ChSL, 4) questions, 5) FULL, and 6) questions. The final design in Table 2 is intended to control for the effect of reading in different formats. Table 1. Frequencies by Group Group Frequency Percent Cumulative Percent TXT-SL-FULL 4 18.2 18.2 TXT-FULL-SL 5 22.7 40.9 SL-TXT-FULL 3 13.6 54.5 SL-FULL-TXT 2 9.1 63.6 FULL-TXT-SL 4 18.2 81.8 FULL-SL-TXT 4 18.2 100.0 Total 22 100.0 100.0  The research team developed a Reading Comprehension Scale based on several aspects of reading comprehension and which included multiple types of questions. Expert recommendations about the questions were incorporated and 21 questions were consolidated: eight open-ended questions and thirteen multiple-choice questions. Multiple-choice questions have four options, one of which is the correct answer, while open-ended questions allow the participants to answer in their preferent way of comunication within a time limit and are scored using a rubric. Vocabulary comprehension was assessed as a covariable for reading comprehension. A Multimodal Response Picture Vocabulary Test was constructed to assess vocabulary in 4 dimensions: writing, signing, fingerspelling, and reading. The first three indicators are examined using 54 items and two trials. In each of them, the subject is presented with an image and has to indicate what it represents by fingerspelling, signing, and writing. The reading indicator is assessed using 54 items and two trials, each composed of 4 images and one word. Participants have to match the image to the word. Results The reliability of all instruments was excellent (𝛼=0.928). A score for the Reading Comprehension Scale was computed for each subject in each modality. As expected, the score for FULL is the highest and the score for TXT is the lowest.

4  

Table 2. Mean scores for each modality Modality M SD CI (95%)

Lower Upper TXT 9.77 4.60 7.85 11.69 SL 10.5

5 4.15 8.82 12.28

FULL 12.00

4.27 10.22 13.78

A paired samples t-test was used to compare the Reading Comprehension Scale between modalities. There was a significant difference between the scores of SL and FULL; t(21)=-2.35, p=0.029. The difference between the scores of TXT and FULL was also significant; t(21)=-2.831, p=0.010. In both cases the score of FULL was higher, which suggests that this modality provided better support for reading comprehension, regardless of where in the sequence it was presented to the subject. To account for a carry-over effect between successive applications of the Reading Comprehension Scale, we computed the mean scores by modality for each group (Table 4). Table 3. Mean Modality Scores in Reading Comprehension by Group Group TXT SL FULL TXT-SL-FULL 9.00 12.00 13.75 TXT-FULL-SL 6.80 9.40 12.00 SL-TXT-FULL 11.67 10.00 13.67 SL-FULL-TXT 13.50 12.00 14.00 FULL-TXT-SL 10.25 11.00 11.50 FULL-SL-TXT 10.50 9.75 8.50 Because the sample size is too small to perform meaningful statistical tests, we ranked the modality scores within each group from top to bottom to see what pattern emerged. While the TXT and SL modalities are distributed among the middle and bottom spots of the ranking in all groups, the FULL modality consistently performs better (in all groups except for one). This seems to indicate that the general comparison between modality scores is valid and that carry-over effects are not important. Table 4. Ranking of Mean Modality Scores in Reading Comprehension by Group Group Top Medium Bottom

TXT-SL-FULL FULL SL TXT

TXT-FULL-SL FULL SL TXT

SL-TXT-FULL FULL TXT SL

SL-FULL-TXT FULL TXT SL

FULL-TXT-SL FULL SL TXT

FULL-SL-TXT TXT SL FULL

To explore the effect of vocabulary in the reading comprehension task, a correlational analysis was performed. A composite vocabulary score was calculated for each participant as the sum of the z-scores of each scale in the Multimodal Response Picture Vocabulary Test. The correlations between the reading comprehension scores for each modality and the

5  

composite vocabulary score were calculated (Table 6). They were all significant, positive, and large, which suggests that vocabulary does indeed support reading comprehension. Table 5. Correlations Between Vocabulary Scores and Reading Comprehension by Modality Vocabulary RCS TXT RCS LS RCS FULL

Vocabulary 1

RCS TXT .598** 1

RCS LS .597** .621** 1

RCS FULL .703** .762** .656** 1

** Indicates significance at the .01 level (bilateral). Conclusions The present study tried to ascertain the compared efficacy of two monomodal conditions—text (TXT) and Chilean Sign Language videos (SL)—and one multimodal condition (FULL), controlling for vocabulary and order of presentation, and the impact of these three conditions on reading comprehension in Chilean D/HH children. All the evidence reviewed suggests that multimodality allows D/HH children to have a better understanding of the text. Results showed that, when exposed to multimodality (FULL), comprehension scores were higher than for any of the monomodal conditions. This proves the positive impact of the information supplied by the conjunction of images, adapted text, and Sign Language. These results were achieved regardless of the order of presentation of the condition. We explored the relationship between vocabulary and reading comprehension, a topic which has already been examined in D/HH students (Loeterman, Paul & Donahue, 2002; Luckner & Cooke, 2010). A significant and positive correlation was observed between the results of the Vocabulary Test (Multimodal Response Picture Vocabulary Test) and those of the Reading Comprehension Scale. Considering the results above, the hypothesis advanced in this study can be confirmed. D/HH people, when exposed to a book, achieve greater reading comprehension if the book is presented in a multimodal format, consisting of adapted textual information, meaningful images, and Sign Language. These results are highly relevant to the consideration of successful strategies for D/HH education. The reading comprehension difficulties of D/HH children may not only be a matter of reading comprehension, but also a result of the modalities in which the content is presented to them. It is important to take some limitations of this research into account. The sample size was small (22 D/HH students) due to the difficulties associated with contacting D/HH students. A larger sample would allow a more detailed analysis. Also, the sample presented great diversity in relation to several variables which are known to affect the education of D/HH children.

References Allen, T. (1994). Who are the deaf and hard-of-hearing students leaving high school and

entering postsecondary education? 1994. Retrieved from http://research.gallaudet.edu/AnnualSurvey/whodeaf.php

6  

Daly, A., & Unsworth, L. (2011). Analysis and Comprehension of Multimodal Texts. Journal of Language and Literacy, 34(1), 61–80.

Dowaliby, F. (1999). Adjunct aids in instructional prose: A multimedia study with deaf college students. Lang, H., 4, 270–282.

Gentry, M. M., Chinn, K. M., & Moulton, R. D. (2004). Effectiveness of Multimedia Reading Materials When Used With Children Who Are Deaf. American Annals of the Deaf, 149(5), 394–403. doi:10.1353/aad.2005.0012

Kelly, R. R., & Gaustad, M. G. (2007). Deaf college students’ mathematical skills relative to morphological knowledge, reading level, and language proficiency. Journal of Deaf Studies and Deaf Education, 12(1), 25–37.

Loeterman, M., Paul, P. V., & Donahue, S. (2002). Reading and deaf children. Reading Online, 5(6).

Luckner, J., & Cooke, C. (2010). A Summary of the Vocabulary Research With Students Who Are Deaf or Hard of Hearing. American Annals of the Deaf, 155(1), 38–67.

Luckner, J. L., & Handley, C. M. (2008). A summary of the reading comprehension research undertaken with students who are deaf or hard of hearing. American Annals of the Deaf, 153(1), 6–36.

Marschark, M., & Hauser, P. C. (2012). How deaf children learn. New York: Oxford University Press.

Mayer, R. (1997). Multimedia Learning: Are We Asking the Right Questions? Educational Psychologist, 32(1), 1–19.

Meier, R. (1990). Person deixis in American Sign Language. In G. Morgan & B. Wool (Eds.), Theoretical issues in sign language research, Vol. 1: Linguistics (pp. 175–90). Chicago: University of Chicago Press.

Mich, O., Pianta, E., & Mana, N. (2013). Interactive stories and exercises with dynamic feedback for improving reading comprehension skills in deaf children. Computers & Education, 65, 34–44. doi:10.1016/j.compedu.2013.01.016

Nikolaraizi, M., Vekiri, I., & Easterbrooks, S. R. (2013). Investigating deaf students’ use of visual multimedia resources in reading comprehension. American Annals of the Deaf, 157(5), 458–73. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/23505841

Paul, P. (1998). Literacy and deafness: The de- velopment of reading, writing, and literate thought. Needham Heights, MA: Allyn & Bacon.

Supalla, S. (1991). Manually Coded English: The modality question in signed language development. In P. Siple & S. Fischer (Eds.), Theoretical issues in sign language research (Volume 2: Acquisition). Chicago: University of Chicago Press.

Wilbur, R. (2000). The Use of ASL to Support the Development of English and Literacy. Deaf Studies and Deaf Education, 5(1), 81–104.