Multisensory books for blind and sighted children
LOLA: Hello, everyone. My name is Lola Chennaz. I'm a PhD student at the University of Geneva of psychology, more precisely, I'm working in the laboratory of sensory motor, effective and social development. And we'll spend about 20 minutes together today during which my colleagues and I will present our research project developments "Multisensory Books for Blind and Sighted Children." So let's do this. To begin this adventure. I will present you the different members of the project. Firstly, this project is carried out by the French nonprofit publishing house Les Doigts Qui Revent, in English, the dreaming fingers, which is specialised in the addition of tactile illustrated books. Sophie Blain is the director. Three laboratories are participating in this project too. A laboratory of technology trait which is human and artificial cognition. And especially the THIM team which is technology handicap interface and multi modality where members are developing researchers about assistive technologies and accessibility in all its forms. Since more than 20 years, Professor Dominique Archambault has been working on projects about accessible audio and tactile computer games and audio and tactile books in collaboration with the publishing house, Les Doigts Qui. And then we have tubes psychology labs, the laboratory of development, individual processes, handicap and education, DIPHE at the University of Lumière Lyon 2, with their lecturer Anna Rita Galiano and the director, and then Dannyelle Valente as a lecturer. Then we have the laboratory of sensory motor affective and social development SMAS, at the University of Geneva. Edouard Gentaz is the director and the professor and myself as a PhD student. These two psychology labs are developing research in the field of touch, visual impairment and comprehension of tactile books by blind people. Speaking of tactile books, what is a tactile illustrated book? They are devices that combine text, braille and large prints and tactile illustrations. Here in the picture, you can see the sun, a caterpillar and the fruits, which are tactile illustrations made in a certain textures. The rail and the writing and the large prints. The idea is just to have a book accessible to all people regardless of their visual status. How important are tactile illustrations? Being the fact that these books help accessibility for all, they're also important for other reasons. Thus Miller, Stratton and Wright have showed that as sighted children, the presence of tactile illustrations in the book is beneficial for the development of language and emerging literacy. Moreover, with Bara, Gentaz and Valente, we observed that illustrations also play a role in the child's understanding and memorization of the story and add fun and playfulness to the learning process. Nevertheless, the pleasure felt by the blind child, discovering a tactile image is different in nature from the image feeling of pleasure of a sighted child perceiving a visual image. In classical design tactile illustrations are transferred from visual illustrations only. Different techniques are used. The techniques of textures on the rights on the picture, the technique of the thermoform, in the middle of the picture, and line drawings. The technique on the right on the picture. Here the lion is represented in profile with its full legs its tail its head with its mane. This image represents the overall shape of lion as usually found in albums for sighted children. But the question is, is it understandable for children with visual deficits? In order to answer to that question, let's talk about studies on the recognition of tactile illustrations transferred from visual informations with blind children. Several researchers have shown that content's based on visual reality can be difficult to understand for children and adults with visual impairments. They explained this difficulty by punting three principle factors. Number one, the lack of familiarity with the visual conventions of drawing. Number two, the specificities of touch. In fact, the haptic sense is very efficient in the perception of the textures and hardness of materials but it is less so regarding visual special qualities of objects represented in these images. And three, the limited ability to generate the visuospatial imagery. So to illustrations exploring other senses. These results invite us to rethink classic procedures of simply highlighting visually illustrations in tactile devices. So the question is, what are the ways of illustrating objects based on other perceptual experience? Such as touch, sensory, motor and sound be more appropriate than visual spatial representations of objects only available to sighted people? And would other properties of objects be more prominent to the blind context? Therefore in order to respond to these questions SMAS and DIPHE psychology labs were interested in development and testing of haptic illustrations. As haptic perception comes from the individuals active experience of the environment. It includes all the tactile and connects the experiences of the body in action. Thus, researchers thought about other ways of illustrating objects based on other senses and therefore not only touch but also sound and sensory motor experiences. In particular, we conducted two studies to test a new design of haptic illustrations, 3D miniatures that children explored using their fingers to simulate leg movements. Here on the is screen an example of 3D miniatures of a trampoline that was explored with both fingers by simulating two legs bouncing on it. And the 3D miniatures of a slide that was explored with both fingers by simulating the movement of sliding. Following embodied approach, our main impetus was that simulations of haptic and sensory motor experiences of the body good facilitated the recognition of illustrations by blind children, because it was reactivating the motor components involved in interactions with the miniature objects. These two studies were conducted in partnership with Litwak, who have made the prototypes. Instead of one, we examined whether the simulations of actions performed on objects, using gestures here the actions simulations by fingers gestures held the same symbolic meaning to blind and sighted individuals. For this, we asked blindfolded sighted, early blinds and late blind adults to produce action simulations by fingers, gestures of 18 action concepts. For example, going up the stairs. Results show that blind and sighted adults had the same motor patterns when simulating several actions. In study two, we examined the densification of seven 3D miniatures of action objects by eight early blind and 15 sighted children, age seven to 12, by exploring them through the procedures of ASFG. Results were compared to another experiment, a control one, with a eight different early blind children and 15 sighted children, age seven to 12 too. They're asked to identify the same action objects depicted in two dimensional by your usual technique of texture. Here on the screen, a clip of the test. Actually, three illustrations. Here, the first clip shows the child climbing the ladder and going down the slide with his fingers without difficulties saying a liner, a slide. On the second clip, the giant finds the trampoline and jumps on it with his fingers and makes the noise. Secondly, 2D illustrations. It's difficult for the girl who found out that she is exploring a swing. She says, "Oula, what does that remind me of?" Here, the girl easily finds out that it's a slide with a ladder. Results. I've shown that objects were more easily and quickly recognised in 3D illustrations engaging simulations than to detect chart illustrations. In addition, the difference of illustrations recognition between blind and sighted children was much less important with 3D illustrations. Thus results provided scientific evidence of the role of embedded cognition and sensory motor experiences in the identification of illustrations by blind and sighted children. It also means that these illustrations have a very inclusive potential to be relevant for a larger number of subjects regardless of their visual skills. Ah, there it is encouraging results. The publishing house Les Doigts Qui Revent has evaluated and you prototype book combining tactile elements that refer to the experience of the body and sound. There's prototype those included illustrations with tactile sensors triggering sounds such as some atmospheres, realistic noises and was associated with a digital tablets allowing to adapt the legibility criteria of the story text, such as phone size and offered it's sound amplification and then the electrical power source to the tactile book. Yes, the idea was to create a more immersive experience for blind and visually impaired children while studying the relative peace of the body in the reading experience. This book prototype was created by the provider Fabernovel based on three pages of the tactile album of work you have called, "La chasse a l'ours," "Beer Hands" in English. Here, clip of the tests The child is reading braille, Oh, a meadow. When he hears the sounds of nature. Here the child is discovering illustration of a river with the sounds of water. He says, Oh, it makes the sound of water now he says he's diving into the river while making the gesture with reason with a big smile. The results of the test of the prototype showed that in terms of economics, it was difficult to ask the child to manipulate simultaneously or alternatively a tactile book and the tablets, especially in the context of a fun reading activity. From then on lead like you have had to think about an alternative that would allow them to choose one of the two sports without altering too much. The technology technological intelligence of the first prototype, moreover to study the relative place of the body. In reading experience, two sessions of reading workshops with groups of blind and visually impaired children where carry out a demand to care institutes in France. We asked the children to carry out different types of action, everyday actions such as walking, opening a door less regular or never experienced actions, such as pushing back to grass and social interaction actions such as nodding your head in the Effie motive. You observe the results showed that the group of blind children used action more naturally to interpret the story of the book thus the action and gestures allow them to enter into the imagination and experience the situation. Whereas the group of visually impaired children tended to intellectualise the action more and just imagine it in their heads without doing it. In conclusion, these field experiments with our research results on the recognition of 3D illustrations with gestures shows us that the body has a high importance in the understanding of the story and especially for blind children. Therefore our current project is based on the combination of these different, with the goal of creating and testing books that activate multiple senses by combining haptic illustrations, such ASFG and OGU experiments. The creation of these books is done with several specialists in different fields of psychology research laboratories, defence mass publishing house for design lead work you have and the laboratory of new technologies and handicap teams. The recent development of programmable components and the ecosystem of tiny active components and send source allows us now to embed some complex electronics in the books themselves. At the beginning of this project we designed the first prototype and titled purchase. Most supplement in English Leedle hand takes walk in which electronic components where incentives in the pages and managed by your tiny digital processor. The processor was embedded in the back cover of the book and itself connected to an external device. Actually it was a computer. Within the pages, we used several sensors, magnetic ones and tiny magnets for the texting. If a page is open and which one also to play with objects that the child can move over the page, smart textiles like some textile of which the resistivity changes when it's folded. And so professors of corporate used as a capacity of sensors to check if a particular place was touched. The corporate could be covered by a thin layer of paper or material yellow provided this material would not be too much insulated for test purpose. We decided to connect the processor to a computer via USB cable in order to get some data from the tests. The computer was also actually playing the sounds according to the story and the action detected and pages the computer was recording any action detected with a timestamp. For example, when a page was turned or a zone touched the use of an external computer was a choice decided mainly for the purpose of collecting data. The process or seems not sufficient to process the whole book including managing the scenario and playing the sounds. But it would have some interesting alternatives with or without an external dedicated device. We tested this prototype with 12 sighted, seven blind and five visual in bed, children aged five to 12. The prototype, Petite main se preme, contains four illustrations. Each one requires different handling that we can include in a tactile book. Each illustration was triggering a sound when they were explored and an knowledgeable instruction was given by recorded voice, which was also written. Here, I will show you clips of the pages with instructions. On page one, the instruction was you to climb the stairs with your fingers. Here we apply the pressure of working with your fingers that we have already studied. The child is hearing footsteps as he walks up the stairs with these fingers. In page two, the instruction was caress the right flowers. Here, we have a more classical pressure of texture exploration. The child heard the sound of nature country noises and birds. (motor engine revving) (birds chirping) Page three, the instruction was knock on the door, knock, knock, knock, and enter. Here we have a pressure jar to simulate the manual action. Here opens door. The child heard the sound of the door opening and the door slamming shots. (door slamming) On page four. The instruction was, 'Oh, A little mouse quick hide it so that the cat can't find him.' Here the pressure jar is a movement of miniature objects. The child was unhooked the mouse from the scratch at the bottom rights of the page and hang it on the top left under cache. The child heard different noises of the mouse. (mouse squeaking) The action was completed. The child heard the mouse thanking him, "Thank you for your help you saved from the cat." We analyse individual reading sessions of the book by the children, visually impaired children performed the experiment blindfolded. At the end of the reading the experimenter asked questions about the action performed and the sounds heard. We then ask the child what he had understood on each page. And finally question with ask the child's general appreciation of the book. The sessions were filmed, and they're recorded by two independent judges. Each child obtained the score for matching between the instructions and the expected, tactile exploration. The number of actions and sounds we called and the comprehension of the illustrations. Preliminary results showed that there was no significant effect a visual status on matching scores, action, sun recall and illustration comprehension. In addition, how you sound action and matching remainders were found for each page. On the other hand, the demand for help was higher on some pages. Finally, the children showed very positive reactions after reading. Therefore, the visual experience does not seem to be necessary for the realisation of the exploratory procedures expected in this book. Moreover, children demonstrated a very good understanding of the actions performed. And the sounds heard as hectic illustrations combined with sounds created a meaningful and playful reading experience for blind and visually impaired children. However, further testing with more blind and visually impaired children in a group of control of sighted children these needed to refine these results. Based on the results of the tests of and the discoveries on haptic illustration. From our previous research, we are working on the creation of a haptic in some book that is easy to use at school and in family. The haptic scenario of this book will be based on the working fingers approach and we'll use the gestures tested and validated by researchers such as jumping turning on a turnstile, flying, climbing stairs. As a publisher, Les Doigts Qui Revent is seeking to design a book that reconciles literary and artistic criteria with optimum accessibility criteria. They're going to mix robustness is of used by children and parents and with economy constraints that will be durably compatible with the construction of a collection whose distribution will be as wide as possible via traditional distribution channels. Finally , there will be an artistic point of view so that the gestural instructions given to the child as sub theme and incubate mixed with the story. The challenge was proposed to the French illustrato, Lucie Felix. Books created by Lucie narrative poetic with experimental multi sensory objects. (indistinct) a book for a site of children where the main character is your hand. This approach is in dialogue with the working fingers approach developed by researchers in the field of tactile illustration from blind children. At the moments haptic scenario and story, is being co-created by Lucie Felix, Dannyelle Valente and the team Les Doigts Qui Revent. At the same time a team in new technologies and interactive design is working on the technical aspects related to the prototype. New test sessions are scheduled with the children's reevaluates comprehension and reading experience. Our presentation ends here. Thank you very much for your attention.