Til hovedinnhold

Video transcription Florence Bara

Combining tactile and auditive information to illustrate storbyook

FLORENCE: This presentation will focus on the value of multimodality, on word memorization and comprehension. The results of two studies will be presented. In the first study, we examined the effect of combining touch and sound on animal names memorization with two children with visual impairment and language disabilities. In the second study, we investigated how tactile illustrations in storybooks can be improved by adding sounds. Shared book reading with children from an early age is a solid grounding for literacy acquisition. Numerous studies have shown that this practice makes easier the development of print awareness of vocabulary and effects comprehension. The shared experience of reading supports children's improvement in literacy outcomes, as well as their interest and their pleasure in reading. Most of children's stories are illustrated. these restrictions are not only esthetically pleasing but would also be useful for comprehension and appetite for reading. Because of low exposure to the variety of written material that surrounds the sighted child environment, children with visual impairment have less opportunity to be confronted to literacy. Although the availability of Braille and tactile books has increased in recent years, it is still far below those for sighted children. Besides, when these books are available, there is no guarantee that the illustrations could be easily understood. Indeed, tactile books are usually designed on the basis of visual criteria that do not necessarily correspond to the tactile exploration skills and mental representation of blind individuals. Different activities and techniques are used to create tactile books and allow more or less easy access to the meaning of the tactile pictures. For example, raised line drawings are poorly recognized by children and lead to relatively long identification times, whereas textual pictures let- lead to better performance. In raised line drawing, the figure and the background can be easily confused. This ambiguity can be removed by proposing fully and best picture. The addition of textures that differentiate the parts of an object and relate them to the tactile properties of the object should help even more overcome this difficulty. The use of miniaturized objects has proved to be an interesting technique for illustrating stories as well. In order to properly perceive the environment, the different sources of information must be combined through a multisensory integration process. Disintegrated perception is more informative and allow the complex nature of objects to be captured faster and more accurately, than uni-sensory perception. When the sensory modality is reduced or absent, as is the case in visual disabilities, compensatory mechanisms happen. Because of these mechanisms, blind people have an increased and accurate use of auditory systems that complete haptic perception and enhance its efficiency. In the present study, we examined the effect of combining touch and sound on words memorization. It is a case study with two students with visual impairments and language disabilities. We supposed that adding sounds might allow better memorization of words. The participants were two male students aged 14 years old. Their names have been changed. Both attended a specialized school for children with associative disabilities. Arthur is highly visually impaired. He has autism spectrum disorder. He does not use braille. He can read in black if the documents are adopted in Arial, font size 72. His academic level corresponds to second grade for reading abilities and to third grade for mathematics. Jules is blind from birth, he has an intellectual development disability, his academic level corresponds to second grade. He uses Braille. A memorization task was administered to the students, they had to recall a list of 10 words immediately after listening to it- to it, and the day after. Three lists of 10 animal names were presented in each condition. The students performed three experimental conditions: a tactile condition where the students manipulate, miniaturized objects that correspond to the words, as you can see in the picture, when, where you see one of the students touching with both hands a miniaturized hippo. A sound condition where the cry of the animal was associated to the words and the sound, plus tactile condition, where the sound was combined with the manipulation of the miniaturized animal. These three experimental conditions were compared to a controlled condition where the words were only listened to before being recalled. And you can see an example of the list of words: fish, giraffe, elephant, hen, crocodile, horse, pig, camel, birds, and rabbits. So, the students are to recall the ten animal names immediately after and in the delayed recall the day after the learning. As the two figures show. In conclusion, multisensory integration process facilitates memory encoding and recall. The combination of sound and tactile information seems to be relevant to help children with visual impairments to memorize a list of words. In this study, we have shown the value of combining sound and touch to memorize words. But what about comprehension when reading illustrated tactile books? This is the purpose of another study that I am going to briefly present. This study focuses on design and evaluation of an interactive tactile book which contains textural images, miniaturized objects, and sounds. We use a user-center ergonomic approach, which seems really adapted to create books accessible to blind people. This approach is based on an analysis of the individual functioning and behaviors in these environments, as well as the identification of these or other needs, and difficulties in order to provide adaptive tools. Three phases were implemented and analyzed. The first is the analysis of the needs and activities of the users with the books. The second deals with the design of the prototype and of the book after ideas iteration. And the third phase is the finite project assessment. The user... Um, becomes part of the entire process of conception and assessments and is involved in each step of the process. Seven children participated in the study, four with visual impairments and three early blind. A narrative story was co-written with specialist teachers, the topic of the text was about an Inuit blind child that was able to help his village thanks to her auditory skills. The prototype of the book combined tactile illustrations and sounds and was designed according to the criteria of cognitive academics. Five sounds were chosen to be added. They correspond to the sounds of wind, fire, knocking, water, and cries of joy and clapping. This sounds were added to the tactile illustration. Um, I will only present the results concerning the effect of adding sounds. All sons except the knocking sounds were immediately recognized by all children. Moreover, as the sounds helped to give meaning to the illustration and support text comprehension, for example, whereas after first a free, active exploration, only one child had identified the fire. After listening to the sound of crackling, all children were able to identify that the illustration depicted a pile of wooden pieces. The water was never mentioned during Arctic exploration of the illustration until children heard it sound. On the contrary, the knocking sound led the children on the wrong track and led them to make additional proposals that were completely disconnected from the context and the content of the story. Consequently, this sound was removed from the final version of the book. In conclusion, the addition of sounds when correctly recognized by children allowed to disambiguate the tactile illustration. Multisensory interactions allow a better understanding of information. And this type of multisensory books, especially when designed for and with children with visual impairment, might be an interesting way to promote literacy acquisition. Thank you for your attention.